Handling URL unfurls and crawlers that don't support JS with WebRender

Often, you’ll find that you have a URL that you want to share with friends, but because it’s a SPA/React application, the title is just the base title, because usually, unfurl bots don’t pre-render your HTML.

By default, if a page is not in the cache, unless a user agent is known not to be able to crawl JS, we will issue a “cache miss” and let React render the page.

However, you’ll likely want unfurls to be faster than full-page renders, and you don’t want us to cache the result for Google. This is why we’ve introduced a parameter called unfurlDelay which can be used to render the header portion of your site without having to pre-render the entire thing.

WebRender automatically keeps a list of user agents that require the unfurl functionality, and we’ve already entered all the major messaging platforms in there, so you don’t have to. You can easily test out the functionality by just sharing your webrender URL (your_key.webrender.io/page) to whatever messaging platform you prefer, and seeing if it works the way you’d expect it to.

Simply turn on unfurl delays by adding it to the same JSON node you have your preRenderDelay

  "origin": "https://example.com",
  "cacheTtl": "auto",
  "renderSettings": [
    {
      "path": "/$",
      "unfurlDelay": 1500,
      "preRenderDelay": 3000
    }
  ]
}

Here is the list of platforms that trigger an unfurl, vs forwarding to origin and waiting for it to render the full JS state (since the crawler does not support JS).

ahrefs
aol
apple (for iMessage)
baidu
bing
discord
duckduckgo
facebook
messenger
reddit
skype
slack
snapchat
twitter
teams (Microsoft teams)
whatsapp
yandex