advertools.code_recipes package
Submodules
- 🕷 SEO Crawling & Scraping: Strategies & Recipes
- How to crawl a list of pages, and those pages only (list mode)?
- How can I crawl a website including its sub-domains?
- How can I save a copy of the logs of my crawl for auditing them later?
- How can I automatically stop my crawl based on a certain condition?
- How can I (dis)obey robots.txt rules?
- How do I set my User-agent while crawling?
- How can I control the number of concurrent requests while crawling?
- How can I slow down the crawling so I don't hit the websites' servers too hard?
- How can I set multiple settings to the same crawl job?
- I want to crawl a list of pages, follow links from those pages, but only to a certain specified depth
- How do I pause/resume crawling, while making sure I don't crawl the same page twice?
- How do I use a proxy while crawling?
- XPath expressions for custom extraction