Let's build something to extract all the content from the web. Who said that we should not use all the information available. Let's crawl, crawl and crawl again. We'll stop when there is nothing left.
A Twitter Crawler and News Indexer
TrollHunter is a Twitter Crawler & News Website Indexer. It aims at finding Troll Farmers & Fake News on Twitter.
Find all URLs of a website in a few seconds - Python
Have you ever wanted to extract all the URLs of a website quickly? We'll tell you how! It is hundreds of times faster than crawling all the pages of a website to find all of its URLs.