What HTTP headers googlebot requests?

What HTTP headers googlebot is requesting?

I decided to write a blog post about googlebot's HTTP headers because as far as I know in SEO community, it is a topic which has not been attracted much attention so far. To put it another way, it may be my mistake but I don't remember having seen some blog posts, articles mentioning what HTTP headers googlebot is requesting in SEO blogs.

Please find below some of the HTTP headers googlebot is requesting from an HTTP server from Google Search Console. The screenshots are taken after sending Fetch or Fetch & Render Requests from Google Search Console by selecting Desktop or Mobile: Smartphone  as  User-Agent.

Please note that, the servers of your sites may not receive the same HTTP headers from googlebot. For example, in below screenshots we see Accept-Language HTTP Header set as en-US because the requested server sends Vary: Accept-Language HTTP header to the client  which you may not send from your server.

Googlebot HTTP Headers: Fetch with User-Agent Desktop

Googlebot Desktop HTTP Headers Fetch

Googlebot HTTP Headers: Fetch & Render with User-Agent Desktop

Googlebot Desktop HTTP Headers Fetch&Render

Googlebot HTTP Headers: Fetch with User-Agent Mobile: Smartphone

Googlebot Mobile HTTP Headers Fetch

Googlebot HTTP Headers: Fetch & Render with User-Agent Mobile: Smartphone

Googlebot Mobile HTTP Headers Fetch&Render

Please find below some of the HTTP headers googlebot is sending to an HTTP server meaning actual googlebot crawls with the server responses to requests.

Googlebot HTTP Headers: Request Robots.txt file with GET method

Googlebot HTTP Headers: If-None-Match HTTP request header used in combination with If-Modified-Since HTTP request header and the server response with 304 status code.

Googlebot HTTP Headers: Request a CSS file with GET method

Why knowing what HTTP Headers a crawler requests is important? 

It is important in the sense that when you say to your clients, you will crawl their sites as googlebot crawls then you should be sure of requesting the same HTTP headers as googlebot from their servers. The response information and later data you collect from a server depend on what you request in your crawlers HTTP headers.

Imagine a server which supports brotli and your crawler requestsAccept-Encoding: gzip,deflate but not Accept-Encoding: gzip,deflate,br.It is not known beforehand how the server will respond in that case, not receiving br in Accept-Encoding HTTP header request, since it is very unlikely but it can even respond with 406 Not Acceptable.But hopefully at the end you would not say to your client that there are crawl performances problems on their site cause it is your crawler  which doesn't support brotli and this is neither your client's nor the server's fault.

Another scenario can be; a web page which returns different content based on the perceived country or preferred language of the visitor, in other words the web page returns different content in different languages. Google call this as locale-adaptive pages. Actually it is the case of my blog http://www.searchdatalogy.com  3 different contents in 3  different languages; English, French and Turkish on the same URL can be served from it. If you send Accept-Language HTTP header in French in your crawler's HTTP headers to a website like mine, your crawl will be completely different than googlebot's crawl in most cases. Yes, sure googlebot can set an Accept-Language header in other languages  than English when it does local aware crawling but most of the time the language it prefers to speak is its own language English. You can find more information about googlebot's local aware crawling here https://support.google.com/webmasters/answer/6144055?hl=en . Now, you may be thinking who in the world will request a page in French language from their crawler, this might not be an SEO scenario but a movie scenario. Well, if you were a French citizen and coincidentally, you and your friends have a very bizarre hobby which is to develop and test crawlers then believe me it is very likely that you finish crawling a site like mine in French one day or other.

Accordingly if you were the client; when you would like to hire an SEO consultant, work with an SEO agency, or in case you were the SEO consultant using a crawler in the market as a service to realize your SEO contract, you should ask the SEO consultant,  the crawl service provider what HTTP headers their crawlers will be requesting while crawling your websites. Finally as a client, you should monitor their crawlers HTTP headers which are being sent to your servers while they are crawling your sites to see if your servers are receiving the promised and correct HTTP headers from their crawlers.

Thanks for taking time to read this post. I offer consulting, architecture and hands-on development services in web/digital to clients in Europe & North America. If you'd like to discuss how my offerings can help your business please contact me via LinkedIn

Have comments, questions or feedback about this article? Please do share them with us here.

If you like this article

Follow Me on Twitter

Follow Searchdatalogy on Twitter

Comments

Legal Terms Privacy

Python SEO

Learn Programming for SEO on Dataizer: Learn Python for SEO

Data SEO

Walid Gabteni: Consultant SEO

Vincent Terrasi: Data Scientist SEO

Remi Bacha: Data Scientist SEO

Recent Posts

SEO data distribution analysis 4 years, 6 months ago
87 million domains pagerank 5 years, 3 months ago
SEO Forecasting 5 years, 4 months ago
SEO data analysis 5 years, 5 months ago
BrightonSEO conference 5 years, 5 months ago
HTTP2 on top sites 5 years, 8 months ago
What HTTP headers googlebot requests? 5 years, 11 months ago
Alexa top 1 million sites 6 years, 1 month ago
1 million #SEO tweets 7 years, 1 month ago
SEO, six blind men & an elephant 7 years, 2 months ago
Technical SEO log analysis 7 years, 3 months ago
3 ways for free https 7 years, 5 months ago
Crawl dictionary 7 years, 5 months ago
Https on top sites 7 years, 6 months ago
SEO web server log files 7 years, 7 months ago
Hsts on google.com 7 years, 7 months ago

Recent Tweets