BUbiNG is a web crawler tool for the next generation that was developed by the authors using their previous work with UbiCrawler as well as their ten years of research on the topic.
Platform independence, linear scalability, graceful deterioration in the presence of faults, a very effective task function (based on consistent hashing) for partitioning the domain to crawl, and more generally the complete decentralisation of every task are the primary characteristics of BUbiNG. BUbiNG also has a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl.
There are a bunch of decent tools out there that offer the same array of services as BUbiNG. And it can sure get confusing to choose the best from the lot. Luckily, we've got you covered with our curated lists of alternative tools to suit your unique work needs, complete with features and pricing.
