Majestic-12 : DSearch : Technology

DSearch is a project that aims to build a Distributed World Wide Web Search engine, which will take advantage of the power of multiple home and work computers around the world in a fashion similar to projects such as SETI@Home. Some details on the technology used to make it happen are here. You can also view a rather simplistic presentation made before Birmingham Perl Mongers User Group (UK) on topic of "Building a scalable distributed WWW search engine ... NOT in Perl!" (requires PowerPoint).

Anyone with a compatible computer with broadband connection can join this project by getting free to download and use software. This client software will run in background and crawl pages. Crawled data will then be send back to base for indexing.

Alpha version of the search engine can be found here.

List of 3rd party supported tools and services is here.