1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. robots.image-blocked-undocumented

robots.image-blocked-undocumented

Image disallowed by robots.txt to crawlers with undocumented handling. Level: Warning.

When it is raised

The API's crawler requests /robots.txt on the image URL's origin, follows at most 7 redirects (8 requests), and stops reading once more than 500,000 bytes have arrived. The API tests the image URL's path and query string separately for each of the LinkedInBot, Slackbot-LinkExpanding, WhatsApp, Discordbot and TelegramBot tokens. For each token the API applies the same group selection and rule matching it uses for robots.image-blocked: a group naming the token is used in place of User-agent: *, groups at the same level are combined, the longest matching rule wins, and Allow wins a tie. A token matches a User-agent line case-insensitively, exactly or with a /version suffix, so a group for Slackbot alone does not cover Slackbot-LinkExpanding. A User-agent: * rule applies to any of these tokens that has no group of its own. The check is raised once, listing every crawler for which the image is disallowed, and can appear alongside robots.image-blocked. A robots.txt answering 404 counts as no rules; any other failure means no robots.txt check is reported.

What it does to a shared link

LinkedIn, WhatsApp, Discord and Telegram publish nothing about whether they honour robots.txt when fetching a preview image. Slack publishes a robots page for its crawler but does not state whether its rules apply to a preview image fetch. The check therefore does not predict whether the picture loads on those platforms.

How to fix it

  1. Open robots.txt on the host that serves the image and find the rule that disallows the image for the listed crawlers.
  2. If the block is not intended, remove the rule, or add a longer Allow rule for the image path in the group the API applies to each token.
  3. If the block is intended, no change is needed.
# https://example.com/robots.txt User-agent: LinkedInBot Disallow: /assets/ Allow: /assets/og/

Related checks

Sources