1. Home
  2. Products
  3. Social card preview
  4. API
  5. Checks
  6. ua-diff.crawler-starved

ua-diff.crawler-starved

Copy served to the API's crawler has fewer values than the browser's. Level: Warning.

When it is raised

The API requests the page twice at the same time: with the API's crawler (user agent PolymorphismOGPreview/1.0, or the one chosen with ?ua=) and with a desktop Chrome user agent. Each request follows at most 7 redirects (8 requests). The comparison runs only when both requests get a 2xx response, and never for ?ua=browser. Three values are read from each copy, in the first 300,000 characters of the HTML, outside comments, and compared after trimming whitespace: the title (og:title, then twitter:title, then <title>), the description (og:description, then twitter:description, then <meta name="description">) and the image URL (the first of og:image, og:image:url, twitter:image and twitter:image:src, resolved against that copy's final URL, after redirects; a value that does not resolve to an http or https URL counts as empty). Raised when at least one value differs and, among the values that differ, fewer are non-empty in the copy served to the API's crawler than in the browser's copy.

What it does to a shared link

The rest of the result reads the copy served to the API's crawler, which lacks values the browser's copy has. Facebook's sharing guide documents Open Graph (og:*) tags as what it reads, and X's Cards documentation, since withdrawn, described X reading twitter:* tags with og:* as a fallback. A platform crawler served the same reduced copy would have only the values in that copy to build a card from. When the title differs, the message shows the title in the copy served to the API's crawler.

How to fix it

  1. Fetch the page with curl as the API's crawler and as a desktop Chrome user agent (the two commands in the example), and compare the tags to find the consent wall, bot gate or user-agent rule that serves a different page.
  2. Serve og:title, og:description and og:image in the initial HTML to every user agent, ahead of any consent or challenge step.
  3. Request the URL from the API again with ?ua=facebook or ?ua=x to fetch the page with the user agent facebookexternalhit/1.1 or Twitterbot/1.0.
<head> <meta property="og:title" content="Page title"> <meta property="og:description" content="One or two sentences about the page."> <meta property="og:image" content="https://example.com/card.png"> </head> curl -s -L -A "PolymorphismOGPreview/1.0 (+https://www.poly.io/social-card-preview/)" https://example.com/page | grep -i -E 'og:|twitter:|<title' curl -s -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36" https://example.com/page | grep -i -E 'og:|twitter:|<title'

Related checks

Sources