Apache Nutch is a highly extensible and scalable open source web crawler software project. Nutch is coded entirely in the Java programming language, but data is written in language-independent formats. It has a highly modular architecture, allowing developers to create plug-ins for media-type parsing, data retrieval, querying and clustering. The fetcher ("robot" or "web crawler") has been written from scratch specifically for this project.
Mixnode is a fast, flexible, massively scalable platform to extract and analyze data from the web. Mixnode allows you to thin ...
StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm. The project is under Apache licens ...
Scrapy is an open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet ex ...
ACHE is a web crawler for domain-specific search
ProxyCrawl helps you stay anonymous while crawling the web, web crawling protection the way it should be. Get data for your SE ...
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometim ...