Skip to content
English
Contact us
  • There are no suggestions because the search field is empty.

User Agents and Whitelisting

What are agents and whitelisting?

A user agent is a piece of software which gathers and presents web content to a user. To do this, it uses a user-agent string which is a line of text used to identify itself to the server from which it is requesting a web page. This can be used for many purposes including:

  • Displaying different content to different operating systems – for example, by displaying a slimmed-down page on mobile devices
  • Serving different web pages to different web browsers
  • Gathering statistics showing the browsers and operating systems in use by their users
  • Allowing the agent to bypass registration screens

Allowing a user agent unrestricted access to a web page is called whitelisting.


Why are they important?

Echobox will still work without whitelisting, but its capabilities will be limited. This may mean:

  • Information such as article images, titles or descriptions may be missing from Echobox
  • Article scores will be less accurate
  • AI Messages can't be generated
  • Automatic Time Sensitivity classification will be less accurate

For the best performance, we recommend whitelisting the Echobox, Facebook and Twitter user agents.

 

Echobox User Agent

The Echobox user agent allows us to access all relevant content for your articles, which can then be used by Echobox to generate post previews and optimise your share timings.

Your tech team will need to whitelist Echobox based on our user agent. For details of the exact user agent you will need to whitelist, please contact your account manager or our support team.

 

Facebook User Agent

The Facebook user agent is used by Facebook to capture OG metadata. It is common for websites to only allow access to this data to the Facebook user agent. You can find out more details on Facebook's user agent here.

The user-agent string which needs to be whitelisted for Facebook is:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

 

Echobox and the Facebook User Agent

Echobox will use the Facebook user agent to get OG data, if the website has a paywall or restricted access.

This is because many publishers only provide OG data in their HTML source when requested by Facebook's agent. Your site may do this in an effort to remove any unnecessary metadata when it's accessed by other agents. In all other cases Echobox will identify as itself using the user-agent string provided.

If you want to differentiate requests made directly from the Facebook user agent and those made by Echobox using the Facebook agent, you can add a header (X-Requested-By) when they whitelist the Facebook agent:

user-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

X-Requested-By: Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)

 

Twitter User Agent

The Twitter user agent is used by Twitter to capture OG metadata. It is common for websites to only allow access to this data to the Twitter user agent. You can find out more details on Twitter's user agent here.

This is the user-agent string which needs to be whitelisted for Twitter:

Twitterbot/1.0

 

Echobox is using the Twitter agent to get Twitter data, if the website has a paywall or restricted access.

This is because many publishers only provide Twitter data in their HTML source when requested by Twitter's user agent. Your site may do this in an effort to remove any unnecessary metadata when it's accessed by other agents. In all other cases Echobox will identify as itself using the user-agent string provided.

In this case you will need to whitelist the Twitter user-agent string so we can get the Twitter data.
If you want to differentiate requests made directly from the Twitter agent and those made by Echobox using the Twitter agent, they can add a header (X-Requested-By) when they whitelist the Twitter agent:

user-agent: Twitterbot/1.0

X-Requested-By: Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)