Majestic-12 : DSearch : Technology

MJ12bot, the principal distributed component of the Majestic-12 search engine project is the subject of continuing investment by Majestic-12 Ltd. The results of this crawl are fed into a specialised search engine with daily updates.

The first prototype full text search engine was built in 2006, this was operational for a period, providing a great deal of insight into running a full text consumer focused search engine. The lessons learned were valuable, but while the project was a success from a research perspective, it was not competitive. As a result, it has not operated for some time.

The prototype full text engine contained an index of around 1 billion pages. After assessing the prototype, Majestic-12 came to the conclusion that in order to operate a full text engine in any meaningful capacity, significant steps were needed, in the areas of:

• Enhancing the relevance of search,
• Enhancing the performance of user search,
• Delivering a relevant and sustainable speed of update.

In addition, it was recognised that to run a search engine of any scale, large investments would have to be made in hardware and infrastructure.

As a result of these findings, a number research projects were initiated. These projects included exploring improvements in the quality of crawl, and the building key search engine components such as a link map. In order to facilitate continuing investment into the program, some of the results of research projects were commercialised, the MajesticSEO product launching in 2008.

Investment continued, largely into aspects around performance and update. Majestic-12 was able to achieve a greatly enhanced crawl, with updates on its web scale backlinks index on a daily basis.

MajesticSEO was commercially successful. MajesticSEO provided access to backlinks index using a dedicated, high performance search at MajesticSEO.com.

MajesticSEO facilitated a scaling of Majestic-12, with more software developers brought on to facilitate a faster pace of development. Significant enhancements were made to the backlinks index, incorporating language detection, heuristic topic analysis, and valuable metrics and potential ranking signals.

In 2014, The MajesticSEO website rebranded as "Majestic" - representing a milestone in the understanding of search as the data became useful to a wider audience than those engaged in traditional SEO activity. MajesticSEO remains a valued trademark and commercial identity.

Majestic.com continues to serve Digital Marketers, Domain Name brokers and online media professionals with analysis and data at web-scale.

Majestic-12 continues to develop a greater understanding of web scale full text search, and demonstrate that understanding through tools and technological developments.