Welcome to Majestic-12!

30/09/05 370 mln URLs added

Just over two weeks from the last URL load more data had to be parsed to catch up with the new high crawling levels with 370 mln new unique URLs added to the system taking number of known URLs to almost 1,800,000,000 unique URLs!

27/09/05 20 mln URLs crawled today!

Today the community set new record worthy of remembering -- 20 mln URLs crawled representing whopping 433 Gigabytes (almost half a terabyte!) of raw data in less than a day! Screenshot of those who made the record is here.

26/09/05 Search engine updated to v0.2.0

Significant update to the search engine fixing a number of bugs and introducing new ability for users to create their own ranking formulaes and see how they change search results!

20/09/05 MJ12node v1.0.8

New version of MJ12node released with bug fixes, better URL scheduling strategy that should minimise number of buckets with 1 domains and lots of URLs left, and a new feature allowing to compress crawled data 15-30% better (not enabled by default, see Options->Crawler->Enable barrel sorting). Details on what's changed are here.

12/09/05 270 mln URLs added

Over 1.5 TB (TeraBytes!) of crawled data was parsed for new URLs, and after deduplication and filtering just over 270 mln more URLs were added to the system taking number of known URLs to over 1,400,000,000 (for those who get dizzy from all those zeroes -- that's 1.4 billion!)

5/09/05 1 bln crawled URLs + MJ12node v1.0.7

Today we reached a major milestone of 1,000,000,000 crawled URLs! It only took 8 months from public beta of the crawler, and next billion should take a lot less time! :)

New version of MJ12node released with lots of changes making it a worthy upgrade. Details on what's changed are here.

29/08/05 MJ12node v1.0.6 + more URLs + Firefox search plug-in!

New version of MJ12node released with an number of bug-fixes and one important change that calls for speedy upgrade of your clients, please get updated versions as soon as you can!

115 mlns more URLs were addded to the system taking number of known URLs to almost 1,150,000,000.

Anyone with Firefox (and why shouldn't you use the best browser available?) can now add Majestic-12 search engine plug-in to the list of search engines, don't be shy to use it to search -- this will help improve the search engine!

28/08/05 Stats upgraded!

Upgraded stats on this site now show breakdown by countries, platforms, best daily rates, more per peer stats and instructions on what HTML you need to use if you want to put some of your personal stats onto your webpages (more ideas in this area particularly welcome). Best place to start exploring new stats is from here.

16/08/05 Search engine updated to v0.1.5 + 1 bln URLs known!

Significant update to the search engine now with almost 45 mln URLs, some bugs fixed and new features such as domain clustering (site: prefix works now). This release will serve as the basis for work to improve relevance of search matches. The search engine is now hosted on a dedicated box so response time should be more reasonable now.

Separately a significant milestone of over 1 bln known URLs have been reached! In light of considerably higher crawling rates than ever before a special effort will take place to ensure that there is enough supply of URLs to crawl without having to panic and add them at the last minute! :)

27/07/05 Talk at Birmingham's UK Perl Mongers group

Presentation of talk before Birmingham Perl Mongers User Group (UK) on topic of "Building a scalable distributed WWW search engine ... NOT in Perl!" (requires PowerPoint). :)