Crawl Dictionary

Crawl Rate

Requests per second a crawler makes to a website when it is crawling it.

Crawl Budget

An allocation of crawl requests to a host.

Crawl Frequency

Program determining which sites to crawl, how often, and how many pages to fetch from each site.

Crawl Rank

The frequency a page is crawled by a search engine bot compared to the ranking position of that page on that search engine.

Crawl Space

The totality of possible URLs for a website.

Crawl Ratio

The number of pages crawled by a search engine bot compared to the total number of available pages to crawl on a website. 100% means search engine knows all the pages on that website.

Effective Crawl Ratio

The number of specific pages crawled by a search engine bot in the crawl window of that type of pages on that website for that search engine compared to the total number of available pages to crawl on that type of pages of the website.

Crawl Window

Timeframe a search engine accepts to send visitors to a URL after it crawled it. That period will vary depending on the type of pages. Knowing that number allows to estimate the Effective Crawl Ratio.

Crawl Depth

Depth is the shortest path ‘minimal number of clicks’ from the homepage to a particular page. Crawl depth is how deep a crawler is programmed to explore a website. 

Crawl Waste

Pages available to crawl on a website which have no unique content, no SEO aim, no add value neither for users nor for search engines.

Crawl Simulation

Crawling not the website but the crawl of that website which is already performed.

Crawl Efficiency

The number of useful crawled pages by a search engine bot compared to all crawled pages by the same bot in a defined period.

Crawl Retention

Crawl timeframe by a search engine bot before a visit is recorded from that search engine.

Crawl Optimization

Intelligent use of Crawl Budget(Allocation) on a website.

Crawl Performance

The average time a crawler spends downloading a page (in milliseconds) 

Useful Crawl

Crawled pages of a website by a search engine bot which bring at least one visit from that search engine in a defined period.

Useless Crawl 

Crawled pages of a website by a search engine bot which bring no visits from that search engine in a defined period.

Simulate Empty Crawl

URLs which are known and kept by a crawler but not requested from the host, typically URLs blocked by robots.txt of a website. A crawler is sometimes configured on purpose to perform that kind of crawl to perform analysis on the links of a website.

Partial Crawl

Crawling specific, selected parts of a website.

Unique Crawl 

Unique URL, a crawler crawls on a single website in a defined period.

Have comments, questions or feedback about this article? Please do share them with us here.

If you like this article

Follow Me on Twitter

Follow Searchdatalogy on Twitter

About Us

Our objective is bringing all our experience and expertise together to deliver solid technology solutions that can take your search traffic acquisition to the next level. Our main goal is to assist you in building and maintaining your search marketing analytics platforms. Our will is to leverage your marketing and IT teams search knowledge while bridging the gap between two.

Legal Terms Privacy

Recent Posts

BrightonSEO 5 days, 10 hours ago
1 Million #SEO Tweets 7 months, 2 weeks ago
SEO, Six Blind Men & An Elephant 8 months, 2 weeks ago
Best SEO Conferences In 2017 8 months, 4 weeks ago
SEO Hero 2017 9 months, 2 weeks ago
3 Ways For Free HTTPS 11 months ago
Crawl Dictionary 11 months, 3 weeks ago
Bing Strikes 1 year ago
HTTPS On Top Sites 1 year ago
SEO Web Server Log Files 1 year, 1 month ago
SEO, Web Server Logs And Science 1 year, 1 month ago
HTTP2 On Top Sites 1 year, 1 month ago

Recent Tweets