Majestic-12 : DSearch : Technology
MJ12bot, the principal distributed component of the Majestic-12 search engine project is the subject of continuing investment by Majestic-12 Ltd. The results of this crawl are fed into a specialised search engine with daily updates. A full explanation follows.
The first prototype full text search engine was built in 2006, but has not operated for some time.
The prototype full text engine contained an index of around 1 billion pages. After assessing the prototype, Majestic-12 came to the conclusion that in order to operate a full text engine in any meaningful capacity, significant steps were needed to enhance the relevance of search, performance, and speed of update. In addition, it was recognised that to run a search engine of any scale, large investments would have to be made in hardware and infrastructure.
As a result of this, research projects were initiated to improve the quality of crawl, and to build key search engine components such as a link map. In order to facilitate these projects, some of the results of research projects were commercialised, the MajesticSEO product launching in 2008.
Majestic-12 now operates a greatly enhanced crawl, with updates on its web scale backlinks index on a daily basis. This backlinks index is open for queries using a dedicated, high performance search at MajesticSEO.com. Majestic-12 continues to offer webmasters the ability to download data for their own sites for free via MajesticSEO, and continues to invest in the improvement of its crawler and search infrastructure.
Majestic-12 continues to make strides towards developing an understanding of the architecture required for effective full text search, and continues to develop the components required for a quality web scale full text search.