Show HN: Building a web search engine from scratch with 3B neural embeddings (blog.wilsonl.in)

Article summary

The author built a web search engine from scratch in two months using 3 billion neural embeddings, with the goal of creating a better search engine that can understand intent and provide high-quality results. The search engine uses a combination of natural language processing and machine learning to index and retrieve web pages. The author discusses the challenges of building a search engine, including handling noise and irrelevant content, and presents a high-level architecture of the system. A live demo of the search engine is available.

Main themes

• Search engine development
• Neural embeddings
• Natural language processing
• Web crawling
• Information retrieval
• SEO and spam

What commenters say

• The internet has changed over time, and a lot of content has disappeared, making it harder for search engines to provide relevant results.
• Google's search results have become poor in recent years due to the company's prioritization of profit over search quality.
• Building a search engine that can understand intent and provide high-quality results is a challenging task that requires a different approach than traditional keyword-based search engines.
• The use of large language models and neural embeddings can improve search engine results, but may also introduce new challenges and biases.
• The web is becoming increasingly dominated by SEO-optimized content and spam, making it harder for users to find relevant and high-quality information.
• A search engine that can provide a 'missing' tab or archive of dead links could be useful for users who are trying to find content that is no longer available online.
• The effectiveness of a search engine depends on the quality of its training data, and using curated and well-filtered data can improve results.
• Some commenters believe that the web is 'dead' and that search engine optimization has become more about manipulation than providing relevant results.