BUbiNG is a web crawler tool for the next generation that was developed by the authors using their previous work with UbiCrawler as well as their ten years of research on the topic.
Platform independence, linear scalability, graceful deterioration in the presence of faults, a very effective task function (based on consistent hashing) for partitioning the domain to crawl, and more generally the complete decentralisation of every task are the primary characteristics of BUbiNG. BUbiNG also has a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl.
There are a bunch of decent tools out there that offer the same array of services as BUbiNG. And it can sure get confusing to choose the best from the lot. Luckily, we've got you covered with our curated lists of alternative tools to suit your unique work needs, complete with features and pricing.
Written by StartupStash
Editorial team
StartupStash is an editorial team. Each Top Tools shortlist is researched by a writer who works in that category, then checked against first-party product pages and public prices before it goes live.
Reviewed by Manaal
Content Manager, Startup Stash
Reads the shortlist, the priced claims, and the sourcing before a piece goes live.
How this page is checked
We read first-party product pages, dated changelogs, and AWS Marketplace when a listing exists. Catalog counts stay vendor claims.
