Botify / Fifty shades of crawl

Fifty Shades Of Crawl

Botify won the Best Search Software Tool Award at the European Search Awards 2016. As someone who knows the company's past very well, I am elated about this win. Botify is a French IT startup which provides advanced analysis of crawl, log file, Google Analytics on big volume of data as a software service, SaaS to more than three hundred customers, such as Expedia, Ebay and BlaBlaCar. By collecting $7,2 million in investment funds, it is now in the top 10 French technology companies with the highest amount of investment funds in the first third of the year.

I had the chance and the privilege to work with many of Botify’s founders and employees, both in IT and sales teams, at their former company Aposition. This small French company, known to be the most expensive in its field, had as its customers the world's largest websites, of which very well-known brands such as Priceminister, Allocine, Pagesjaunes, Dailymotion and Ebay. Its power derived from two genius computer scientists; Stephane Chauvin (now Botify IT Architect) and Jerome Spiral (now Rive World Founder and COO).


If we can talk about a company's religion in that case Aposition’s religion was science and if science was its religion, then crawl was its adulation. What is interesting to point out is that while other SEO companies were and still are very heavily occupied by keyword optimization, yet not much changed today in this regard, what we were doing was crawl optimization. If the first thing the majority of SEOs did in a project was to find out which keywords to target, what we did first was to crawl the site. Huge websites with millions of pages embracing millions of visitors per day did not scare us, contrarily, we considered it a fun task. If others were following the keyword positions they attain on SERPs the same way they would follow daily weather forecast, we were following the crawl of googlebot on the site not only daily, weekly, or monthly basis, but sometimes with even several years of log files. While others were decorating their websites pages with the most desirable keywords, we were sending googlebot wherever we wanted on the site. If half of the words other SEOs used in their speech were keywords, ours was crawl.


Crawl is the main difference, which distinguishes in the past Aposition, today Botify from other SEO companies. We, for whom science is a religion and crawl a worship, never tried to convince search engines that we deserve top positions in SERPs based on beliefs, superstitions, second-hand hearsay information; “in my opinion”, “based on what I read on the Internet” or keyword-targeted micro modifications on the sites. Instead, we did big data analysis. We never began our analysis from the last stop of search which are SERPs, instead, we always began it from the source which is crawl. And therefore, while all the world's SEOs were shaken due to their KPIs which became obsolote when Google passed to “not provided”, we were not affected from that at all because we never based our KPIs on keyword tracking. And maybe that way of analyzing search data explains now the quote from Yoda at the entrance of Botify’s head quarter in Paris; “You must unlearn what you have learned”.

A Crawler

Giant websites need giant crawlers. Botify has a crawler, one of the most powerful in the world. It is holding the world’s record of most URLs crawled from a single website which is 150,000,000. It can crawl 200 pages per second which makes 17,280,000 pages per day meaning it can crawl 100,000,000 pages of a whole website in less than a week. I had once witnessed googlebot crawling 10,000,000 pages per day on a single website but unfortunately I have not had the opportunity to see more yet.

Fifty Shades of Crawl

Clients who worked with Aposition in the past and today with Botify, meet always not only with the conception of the crawl but discover fifty shades of crawl; crawl ratio, crawl rate, crawl speed, crawl statistics, crawl waste, crawl window, crawl retention, crawl efficiency, crawl frequency, crawl freshness, crawl simulation, crawl performance, crawl age, crawl budget, crawl optimization, crawl space, crawl rank, crawl depth, crawl schedule, crawl bandwith, crawl probability, partial crawl, simulate empty crawl, recrawl, unique crawl, useful crawl, useless crawl, first crawl, last crawl, new crawl ...

A Botify Certified Consultant

I feel privileged to have worked for a long time on big, challenging projects with the most talented geniuses who are now part of Botify's dream team. Like a child playing with her favorite toys freely and without any limits, I had the opportunity to have so much fun playing with the advanced tools it now offers as SaaS, years before on powerful servers. Now I am proud to be one of the few hundred happy Botify certified consultants in the world.

This article has also been published at

Thanks for taking time to read this post. I offer consulting, architecture and hands-on development services in web/digital to clients in Europe & North America. If you'd like to discuss how my offerings can help your business please contact me via LinkedIn

Have comments, questions or feedback about this article? Please do share them with us here.

If you like this article

Follow Me on Twitter

Follow Searchdatalogy on Twitter

Related Tags: BOTIFY  


About Us

My objective is bringing all my experience and expertise together to deliver solid technology solutions that can take your search traffic acquisition to the next level. My main goal is to assist you in building and maintaining your search marketing analytics platforms. My will is to leverage your marketing and IT teams search knowledge while bridging the gap between two.


Botify: Botify Certified Consultant

IBM: Data Scientist, Data Engineering Certificates

Google: Google Analytics, Google Adwords, Mobile Sites, Digital Sales Certificated Professional

Coursera: Data Engineering on Google Cloud Platform Specialization

Legal Terms Privacy

Recent Posts

SEO data distribution analysis 6 months, 4 weeks ago
87 million domains pagerank 1 year, 4 months ago
SEO data forecasting 1 year, 5 months ago
SEO data analysis 1 year, 5 months ago
BrightonSEO conference 1 year, 6 months ago
HTTP2 on top sites 1 year, 9 months ago
Desktop & mobile performances 2 years, 1 month ago
Alexa top 1 million sites 2 years, 1 month ago
Best SEO conferences in 2019 2 years, 2 months ago
Web marketing festival 2 years, 9 months ago
Webcampday 2 years, 10 months ago
Queduweb 2 years, 11 months ago
1 million #SEO tweets 3 years, 1 month ago
SEO, six blind men & an elephant 3 years, 2 months ago
SEO hero 2017 3 years, 3 months ago
Digitalzone 3 years, 4 months ago
Technical SEO log analysis 3 years, 4 months ago

Recent Tweets