1. Aysun Akarsu @aysunakarsu SearchDatalogy Crawl-first SEO
2. @aysunakarsu @searchdatalogy #brightonseo Crawl-first SEO
3. @aysunakarsu @searchdatalogy #brightonseo AI-first company Mobile first index SEO Crawl first SEO In SEO we have a priority too!
4. @aysunakarsu @searchdatalogy #brightonseo WWW Crawl Index Query Module & Ranking User Crawl-first SEO focus on what?
5. @aysunakarsu @searchdatalogy #brightonseo Pages present on the site Crawled pages Indexed pages Positioned pages Active pages ROI Crawl-first SEO Why?
6. @aysunakarsu @searchdatalogy #brightonseo 01 02 03 04Understand how Google crawl the site Identify incompatibilities Help Google understand the content Help Google access useful pages Crawl-first SEO objectives
7. @aysunakarsu @searchdatalogy #brightonseo Crawl-first SEO how Lorem 3 Communicate Lorem 1 Crawl data ● Websites ● Search engines Lorem 2 Cross Calculate KPIS DatavizCollect Calculate
8. @aysunakarsu @searchdatalogy #brightonseo Crawl-first SEO tools Crawler Store crawl data & calculate KPIS DatavizWeb server log analyser
9. @aysunakarsu @searchdatalogy #brightonseo Gain knowledge about crawlers Question: Can a crawler have language preference and request a web page in its preferred language?
10. @aysunakarsu @searchdatalogy #brightonseo A crawler A crawler is a Automatically collect data by following links on a webpage. wannabe user Gather information, entertain, shop. wannabe browser Collect data from a webpage.
11. @aysunakarsu @searchdatalogy #brightonseo Store data & calculate kpis
12. @aysunakarsu @searchdatalogy #brightonseo Python R matplotlib ggplot2 seaborn shiny bokeh rbokeh Dataviz
13. @aysunakarsu @searchdatalogy #brightonseo Before crawling
14. @aysunakarsu @searchdatalogy #brightonseo Do pages return different content based on? User agent Perceived country Preferred language Any blocking accesses? Bandwidth of your server? Number of available products? Preferred crawl days/hours? Ask to the client
15. @aysunakarsu @searchdatalogy #brightonseo Google search console Access & collect data Web server logs Analytics tools E.g. Google Analytics Sitemaps
16. @aysunakarsu @searchdatalogy #brightonseo Crawl how many pages per second? Request what HTTP headers? Respect crawl instructions? Robots.txt Source code HTTP Headers Redirections handling Get to know your crawler
17. @aysunakarsu @searchdatalogy #brightonseo https://www.searchdatalogy.com/blog/googlebots-http-h eaders/ Deep dive with googlebot
18. @aysunakarsu @searchdatalogy #brightonseo Request Sample urls with different User agents Geolocated IPs Languages
19. @aysunakarsu @searchdatalogy #brightonseo Google search console Time spent downloading a page(in milliseconds) Get to know the server
20. @aysunakarsu @searchdatalogy #brightonseo Pre-identify Crawl waste Google search console Urls parameters Compare active pages GA Web server logs
21. @aysunakarsu @searchdatalogy #brightonseo Select the right User agent IP Language An efficient crawl configuration Set the optimum Crawl depth Crawl rate Avoid crawling crawl waste Create To keep or not in crawl database links? Blocked by robots.txt To other domains
22. @aysunakarsu @searchdatalogy #brightonseo Inform the client your crawler’s User agent IP Calculate how long the crawl will take If you crawl 10 pages per second how long will it take to crawl 5 M pages of a site? Inform, calculate
23. @aysunakarsu @searchdatalogy #brightonseo Analyze crawl results If all is fine launch your crawl Run a test crawl
24. @aysunakarsu @searchdatalogy #brightonseo Categorization & kpis
25. @aysunakarsu @searchdatalogy #brightonseo Categorization
26. @aysunakarsu @searchdatalogy #brightonseo SEO page types Vestibulum congue Vestibulum congue Available Compliant Crawled Active
27. @aysunakarsu @searchdatalogy #brightonseo Page types on a siteHomePage Category Category SubcategorySubcategorySubcategorySubcategory ProductsProductsProductsProducts Products
28. @aysunakarsu @searchdatalogy #brightonseo Not all queries are equal Homepage Category Subcategory Subcategory Products Products Products Cost&Competition Search frequencyLow High Low High
29. @aysunakarsu @searchdatalogy #brightonseo Main KPIs 5 C raw ldepth 4 Internalpagerank 3 C raw lfrequency A ctive pages ratio 2 6 C raw lperform ances 1 C raw lratio
30. @aysunakarsu @searchdatalogy #brightonseo Structural crawl analysis
31. @aysunakarsu @searchdatalogy #brightonseo Principal universes Structural Crawl Analysis Site Crawl waste Other sites Useful pages for SEO Useless pages for SEO Links to other sites pages
32. @aysunakarsu @searchdatalogy #brightonseo Principal categories Popular pages Listing pages Long Tail By theme Site Homepage Other popular pages Main listing pages Secondary listing pages
33. @aysunakarsu @searchdatalogy #brightonseo How many products ?
34. @aysunakarsu @searchdatalogy #brightonseo How many videos? Client Consultant I have 1,6M videos available on my site We crawled your site to the crawl depth 10, accessed only to 1,4M videos Where are 200 K videos?
35. @aysunakarsu @searchdatalogy #brightonseo How many products? Client Consultant There are 1 M products on my site Out of 780 K pages which are available to the crawler, 600 K are accessed by the structure of site 180 K pages are available outside the structure. Are you sure of having 1M products on your site?
36. @aysunakarsu @searchdatalogy #brightonseo How many articles? Client Consultant There are 2 K articles on my site Out of 5 M pages which are accessed by our crawler, 2 M of them are articles. Your site has serious problems.
37. @aysunakarsu @searchdatalogy #brightonseo How many listing pages?
38. @aysunakarsu @searchdatalogy #brightonseo Consultant Client You have 2 M products on your site but only 20 K listing pages. You should increase the number of listing pages Why? By increasing listing pages, you can increase middle tail traffic. Currently the crawler discovers small part of your product pages by the listing pages. If you increase them, it will discover the totality of products through them which will have a great effect on product pages crawl ratio. When not enough
39. @aysunakarsu @searchdatalogy #brightonseo Consultant Client You have 3M pages available on your site. 1M products and 2M listing pages. You should decrease the number of listing pages Why? Because a search engine can produce statistical analysis to estimate the proportion of original information on a website. With 2 M listing pages presenting 1 M products pages, you will not have good statistical results. When too much
40. @aysunakarsu @searchdatalogy #brightonseo Popular pages?
41. @aysunakarsu @searchdatalogy #brightonseo Consultant Client You have 2 M products that bring 70% of traffic to your site. These pages receive 50% IPR, their crawling ratio is not satisfactory. I recommend you to send more IPR to your product pages to increase long tail traffic. How? There are some parts on the site which don’t deserve IPR they receive. Why not sending less IPR to them? Let’s send some popularity
42. @aysunakarsu @searchdatalogy #brightonseo Thank you Photos on Unsplash by Aaron Burden Nicolas Picard Nathan Anderson Michael Podger Frances Gunn Erwan Hesry Dev Leigh
43. @aysunakarsu @searchdatalogy #brightonseo Appendix
44. @aysunakarsu @searchdatalogy #brightonseo A web server log analyzer https://www.searchdatalogy.com/blog/is-my-s ite-in-googles-mobile-first-index/
45. @aysunakarsu @searchdatalogy #brightonseo A web server log analyzer https://www.searchdatalogy.com/blog/ technical-seo-log-analysis/