The Old Joel on Software Forum: Part 4 (of 5) - Another look at the search engine world

Another look at the search engine world

<begin rant>
Why does Microsoft think it must involve itself in "anything and everything" to survive? Because it probably does have to do that to retain its current influence. And I find myself thinking that google would probably soon be forced to adopt the same strategy for the search market.

First of all, what do people search for?
1) Products they want to BUY: objective reviews, comparative pricing, sites where they can actually shop
2) Reference Information on a particular TOPIC: "Java", "C++", "Airplanes"
3) Websites belonging to particular organizations, people, or projects, product: "Joel On Software", "Joel Spolsky", "Microsoft", "Adobe".
4) Where to download some software: "Lavasoft ad-aware", "Remote Desktop Connection"
5) News about a particular TOPIC: "Iraq prison abuse"
6) Ask a question: "Why is Joel Spolsky so confident of his opinions?" (ok, just kidding)

What other categories do you think I might have missed?

Now, a point is that even though there are specialized search sites/services for most of these categories(1-amazon.com,epinions.com; 2-encarta.com; 3-Yellow pages; 4-download.com; 5-cnn.com,news.google.com; 6-??????), people still tend to go to a general search engine for most searches because:
1) they want 'objective' answers
2) they want choice: answers from more than one website
3) they don't know about the specialized search pages

People talk about personalization as a next step for search engines but they won't define it. The problem of a general-purpose search engine is that it does not know your INTENTION (Amazon.com knows or assumes you want to buy; epinions knows you want a review; cnn.com knows you want news). The solution to that is not necessarily by storing search history or asking me what topics I am interested in.

One form of solution will be some sort of agent that runs on my _desktop_ (here comes microsoft!) and infers from my behavious what my intention is at any time. Then suggests sites or special searches before I have the time to think of going to a general purpose search engine.

The other form of solution, a temporary or fall-back solution where the agent fails, is a search engine that understands the various 'senses' in which a query might be meant and suggests alternate searches that would yield more accurate results. Integrated with software that monitors my behaviour, the search engine should be able to know what 'meaning' is more likely for the query I enter.

Therefore algorithms for the general-purpose search engine must go many steps beyond keyword or link analysis based on raw web pages:
- At its backend must be a host of specialized search engines. keyword-based web search (aka google:P) will be just one of them; the one it falls back on for searches that don't fit into any of the known categories.
- It must at least understand words, phrases and the fact that they have different meanings on different pages. Algorithms to understand context better.
</end rant>

What's your take on the various points?

Seun Osewa (afriguru.com)
Sunday, May 2, 2004

What points?

Jorel on Software
Sunday, May 2, 2004

"Influence"?

Microsoft wants to enter the search space in order to maximize shareholder value. That's what public corporations do.

Ankur
Sunday, May 2, 2004

Corporations are not mindless robots, they are managed people. By "infleunce" I meant "market domination" which, you would agree, really helps Microsoft to provide shareholder value :-P

Seun Osewa (afriguru.com)
Sunday, May 2, 2004