1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. robots.image-blocked

robots.image-blocked

Image disallowed by robots.txt. Level: Warning.

When it is raised

The API's crawler requests /robots.txt on the image URL's origin, follows at most 7 redirects (8 requests), so the rules may come from another host, and stops reading once more than 500,000 bytes have arrived. The API tests the image URL's path and query string against the rules for the facebookexternalhit and Twitterbot tokens. For each token the API applies the most specific group: a group naming the token (matched case-insensitively, exactly or with a /version suffix) is used in place of User-agent: *, groups at the same level are combined, the longest matching rule wins, and Allow wins a tie; * matches any characters and a trailing $ anchors the end. Raised when the image is disallowed for either token. The message names the Disallow rule that decides the result for the blocked token, and the platform; when both tokens are blocked, it names Facebook and Facebook's rule. A robots.txt answering 404 counts as no rules; any other failure means no robots.txt check is reported.

What it does to a shared link

X's Cards documentation, since withdrawn, described Twitterbot as implementing Google's robots.txt specification and treated a blocking robots.txt as the cause of a missing card. Facebook documents robots.txt handling for facebookexternalhit, says the crawler might bypass robots.txt when performing security or integrity checks, and says nothing specific about og:image. The message says the card most likely appears with its title and description and without the picture on the platform named, and for Facebook adds that this is likely rather than certain.

How to fix it

  1. Open robots.txt on the host that serves the image, which may be a CDN or a different subdomain from the page.
  2. Remove the Disallow rule named in the result, or add a longer Allow rule that covers the image path.
  3. If robots.txt has a group for facebookexternalhit or Twitterbot, make the change in that group.
  4. Run the check again; the fix has worked when robots.image-blocked no longer appears.
# https://example.com/robots.txt User-agent: * Disallow: /assets/ Allow: /assets/og/ <!-- the page's og:image tag --> <meta property="og:image" content="https://example.com/assets/og/card.png">

Related checks

Sources