SEO Web Server Log Files

What Is A Web Server Log File?

A web server log file is a file which keeps track of all requests that are sent to that web server. The requests can come from humans through browsers or from bots (like search engine bots) through crawlers. Each requests either from humans or robots produces a single line of code in these files. Web server log files are also very useful to analyze the audience of a website because they provide precise information on site traffic. 

What Does A Web Server Log File Look Like?

What Is The Information In One Line Of A Web Server Log File?

Web Server Softwares and Web Server Logs Format

http://httpd.apache.org/docs/current/logs.html#accesslog

http://nginx.org/en/docs/http/ngx_http_log_module.html


https://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/676400bc-8969-4aa7-851a-9319490a9bbb.mspx?mfr=true

Why Do We Need Web Server Log File Analysis For SEO?

Any analytics tools provide information about the pages bringing visits to a website but none of them provide complete information about the pages which are crawled by search engine bots.
The information given about indexed pages of a website from google  is not accurate, besides it is not possible to access or extract those pages completely.

The only tool that gives information on google crawl is Google Search Console  but ...

Why  Google Search Console Is Not Sufficient For SEO?

In SEO what we focus on is the contents and the strategical contents crawl by googlebot.

1) However at Search Console on crawl stats graph we see the crawl of all the google bots and all web elements crawls (css, javascript, flash, pdf, images) on a  web site.

2) We can not identify the status code of the crawled web pages on that crawl stats graph.

3) The crawl stats graph shows unique crawl but in SEO we would like to know how frequently (or how many times) a certain page or some pages are crawled too.

4) Crawled pages by googlebot can not be accessed or extracted from Search Console therefore we can not

  • Categorize them
  • Analyze them
  • Keep historical crawl data

5) We do not have access to all active pages ( pages receiving organic search engine visits) data. In Search Console we only have access to pages bringing Top and Middle Tail traffic, we do not have access to the active pages bringing in long tail traffic.

Have comments, questions or feedback about this article? Please do share them with us here.

If you like this article

Follow Me on Twitter

Follow Searchdatalogy on Twitter

Comments

About Us

Our objective is bringing all our experience and expertise together to deliver solid technology solutions that can take your search traffic acquisition to the next level. Our main goal is to assist you in building and maintaining your search marketing analytics platforms. Our will is to leverage your marketing and IT teams search knowledge while bridging the gap between two.

Legal Terms Privacy

Recent Posts

BrightonSEO 5 days, 10 hours ago
1 Million #SEO Tweets 7 months, 2 weeks ago
SEO, Six Blind Men & An Elephant 8 months, 2 weeks ago
Best SEO Conferences In 2017 8 months, 4 weeks ago
SEO Hero 2017 9 months, 2 weeks ago
3 Ways For Free HTTPS 11 months ago
Crawl Dictionary 11 months, 3 weeks ago
Bing Strikes 1 year ago
HTTPS On Top Sites 1 year ago
SEO Web Server Log Files 1 year, 1 month ago
SEO, Web Server Logs And Science 1 year, 1 month ago
HTTP2 On Top Sites 1 year, 1 month ago

Recent Tweets