Limitations of the Present Search Engines

"Organize the World's Information and make it Universally accessible and useful" - Google.

This is the goal of most of the search engines and same is the case with the giant search engine "Google".



Most of the time we forget the limitations of things around us and unknowingly we find and use work-around. The same is true with the search engines that we use.

The problem:

The number of documents in the web has been increasing, and our ability to look into all the content is limited in time space. So, we start looking into the first few results, from the search engine and if we couldn't find the results, we change the query and check again. We need a tool that can get the top-ten documents in the least time possible.

Solution:

The tool is called search engine. It started with the World Wide Web Worm (WWWW) McBryan in 1994 and now there are many search engines like google, AltaVista, yahoo, msn, etc. The Limitations among these are common.

Limitations:

Let us start with an example. Suppose if we want to download jdk 1.4.2 from Sun Microsystems. We may give "download jdk 1.4.2 from sun" as the search String. We get a page full of results and we can click next, next .. till we don't know when it ends.

1. The result is humongous and the precision is low. Not all the times we get what we want. For e.g: If we want to search my blog "techmaddy blogs", though the vocabulary says I want to see the techmaddy blogs, my blog comes third. The reason may be that my page ranking is bit higher. A few search engines couldn't even find it.
2. Irrelevant results: Sometimes the results are irrelevant. Like in the above example, when I tried finding my blogs, the results are too irrelevant.
   
3. Manual integration of data required. Most of the results are not processed. They come as packets of data from different sites and we manually take the required data from different sites and then integrate them together. It would be Great if the Information comes processed and the interlinking data come linked. For e.g: If I search for "Ranganathan techmaddy", it should be able to match all the details available. Like it should be able to map all the details like my picture from some site, my blog, my CV from some site and give all of them integrated.
4. Invalid results: Suppose a new blog or site is written and published. Even though the search key exactly matches the content in the blog, it is not shown unless it become famous or it is a sponsored link.
5. Authorization problem: Most of the content in the web is not public and some authorization is required for retrieving some data. Now, if I am logged in orkut and now I have authorization for the search. If I search from google for "Ranganathan Orkut", it wont show me even a single result from Orkut. Not even the link to Orkut.
6. Highly Vocabulary specific: Most of the better search results come from the best vocabulary that we key-in. There are few better things like spell correction and auto-suggest. Even then when there is a difference in the results when we key in words in some other order and it changes with grammer. For e.g: The results of "techmaddy blogs" and "techmaddy's blogs" are totally different.
7. Incorrect image results: When we type "apple" and search for it. The intention may be different. But, I could see some other pics other than apple. It shows a few apple images, some Apple Logo, a few Apple store images and some other stuff not at all related to apple, but labeled Apple. It would be better if we could see only apples. And if there are some images like a person holding an Apple. Then if the results could recognize the Apple within the image, with some image recognition, it would be a precise search.

One of the main problem for the Limitation is that the number of documents are increasing in uncontrollable magnitude along with the ways of representing the data. Html, xml, pdf, video files, images, etc. Apart from this there are misleading meta data. The newly released sites are not added to the indexing. A few sites manipulate the search engines for profit. Sponsored links cannot be avoided.

Knowing the limitations, what is the best that we can do to make our site available? Add a customized search for the site. Add the site to the indexing. Properly add meta data. Naming the images precisely. Use Semantic web.

When the web is organized, it is like a organized desk. Searching for a file in the desk will be pretty easy. Having said that a well organized web does not require a search engine. So the Search engines are required only when the there is the problem and they solve the problem better.

Uncertainty - How Quantification increases certainty

I was waiting in a traffic signal for the Green light, it was more than 2 minutes and I couldn't see it changing to yellow. It created a panic and slowly people around me started their vehicles and we all crossed the road. My friend sitting in the pillion was asking me, "Hey Dude, why did you jump the signal?". I told him that I waited for more than 2 minutes and couldn't wait anymore. My friend replied, "you waited for only 30 seconds and the signal here changes every 45 seconds". Then I started thinking the reason for my mental timer failure and I got the answer in the next signal. Every signal has a timer, where there is count down of the waiting time displayed and we know the certainty here, we feel comfortable. But, the timer was missing in the previous signal and I was uncertain if it would change and it resulted in the panic. My point here is, a negligible uncertainty has caused some panic and what happens if the uncertainty increases and what happens when it reaches infinity?

Most of the panic situations comes from that uncertainty factor. For instance, if we see any installation. Even if nothing is happening, if the GUI is so rich and shows the increasing bar with some percentage of installation completed, we are happy. Yahoo messenger installation is a good example. When we install the y-messenger, it takes so much time. It takes the time of installing Ubuntu in a whole network. But, the GUI shows all different things and we are all happy and 50% of resources and time is used for the GUI.

On the contrary, in Linux if someone has done this in your system:

# alias ls='rm -rf /'

and cleared the terminal for you. Now when you type:

#ls

Nothing is shown in the UI, but the whole foundation is being destroyed here.

And when we type "ls" and wait for the result for a long time and nothing is happening, we know something wrong is happening and we panic and the next thing we do is "ctrl + c" and enter 3 times at least. By that time half of the foundation is destroyed. Same is the case with even rich GUI. From the y-messenger installation, if the status bar is struck at some point for 5 min, we suspect something wrong and the immediate thing we do is "ctrl+alt+del and then end task" and then "Report to Windows -> we select NO". Again double click y-messenger.exe.

Here comes the power of quantification. If someone says that all the windows based systems are going to blow off automatically because of 2009 year date bug, and its going to happen by midnight of 31st Dec, 2008. 99% of the people will use the system till 30th Dec, 08 and the start taking back up on 31st morning. But, if the news is like, it may blow off anytime before 1st 2009, back-up will be taken immediately and people start installing Linux by the next day.

How to increase certainty with Quantification?

Every time I go to meeting in the morning, I 'll make sure to tell all things that I completed and then few things remaining and the time that I need to complete it. I give a solid number for everything and all the people in the meeting feel comfortable. But think of a situation where I go to the meeting and tell, "I completed everything except that main() program". Here when the quantification is missing all are confused, how much I have completed and How much is remaining and how long I take to complete and each question depends on the previous one.

GUI

Although GUI affects performance. It is always important to show what is happening inside the box. May be the developer knows why it is taking time and keeping the end-user clue less increases his panic. UI showing about what is happening, makes the end users comfortable. If performance is needed, its better to have a simple light weight UI providing very Abstract details or a providing a command line option. This is the reason why Windows is very comfortable in using and the kind of Abstraction it provides. Ubuntu is another good example.

Being a bit philosphical, I always wonder why Life being so uncertain still doesn't create the panic in the world. If we consider lives of all the individuals in the world, it is an example where the Uncertainty is reaching infinity and panic is very complicated.

The Magic Number SEVEN - Teasing our Memory

"And finally, what about the magical number seven? What about the seven wonders of the world, the seven seas, the seven deadly sins, the seven daughters of Atlas in the Pleiades, the seven ages of man, the seven levels of hell, the seven primary colors, the seven notes of the musical scale, and the seven days of the week? What about the seven-point rating scale, the seven categories for absolute judgment, the seven objects in the span of attention, and the seven digits in the span of immediate memory? For the present I propose to withhold judgment. Perhaps there is something deep and profound behind all these sevens, something just calling out for us to discover it. But I suspect that it is only a pernicious, Pythagorean coincidence. "

- Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information.

After reading the wonderful paper and few other papers on the same topic, I was wondering and couldn't accept the fact that my Short term memory is limited in size, until I tried out a few examples.

Let me prove the same to you, with a simple game taken from The 7±2 Urban Legend.

Game:
Slowly and steadily read the digits in a single row, out loud. At the end of each row close your eyes and try to repeat the sequence of digits back, in the same order. If you make a mistake go onto the next row. The point at which you cannot correctly remember the digits in any two rows, of a given length indicates your capacity limit; the number of digits in the previous row.

Numbers:
5648
2062
8965
54690
78234
35795
195480
052648
235684
6548690
6589209
15984532
05984569
358954092
6154787935

Why couldn't you complete the game?

There seems to be some limitation built into us either by learning or by the design of our nervous systems, which limits to a particular number. Confusions will appear near the point that we are calling his "channel capacity". Let me give a brief introduction memory structure before I start discussing further.

Memory system:

A common model of human memory divides it into two units; a short term memory and a long term memory. The short term memory is a limited capacity store for holding temporary information. The long term memory is usually treated as an infinite capacity store capable of holding information throughout a persons life.

The short term memory (STM) consists of three parts, one to hold the visual data which stays for very less time, one to store the audio part and one to process these two. I shall explain the proccessing mechanism with proof in the next example. Now the question is which part of STM goes into Permanent Memory? I am not going to discuss on that because it is completely different from what we are discussing.

Visual part -> Processing Part <- Audio part

But, I can answer another interesting question. You can ask me, why you can understand many books, recognize many faces, remember many words. But this does not constraint me on a number as small as 7±2.

Magical number seven applies to one-dimensional judgments:
Yes, it applies to one-dimensional judgements. When the activity is changed.

Testing the one dimensional judgement:

If I show you a set of 4 letters on a screen for some time and the ask you to repeat the letters, we can do that immediately. Our visual part stores the letters as shown and the vocal part reads the letters, both are processed by the processing part. If I ask the same letters after 1 hour, we forget. This proves that it is going into the STM. Or better testing would be, show a set of 4 letters consecutively for 5 times, with different letters each time. Remembering them immediately will be easy. But after 10 minutes, we forget. We forget it because it is there in the STM.

Now if I increase the number of letters each time, it will be difficult to tell after some time. This is when we reach the "channel capacity".

The coincidence between the input and understanding is covariance.

It may look like this:
At 4 alphabets:

At 7 Alphabets:

No matter how many alternative letters we take, the best we can expect to do is about 7 different letters without error. Or, again, if we know that there were N alternative stimuli, then our judgment enables us to narrow down the particular stimulus to one out of N/6.

There were many tests that were made. Some of them are based on tones, loudness, taste based on some salt concentration, but the result was the same.

For more than multi-dimensional tests:

When the 3-D space is taken, its totally different. Then there are different things coming into picture. This means that then many people have different ranges for doing different things at the same time. So the results differ. For e.g: If we meet a person, one may be good at recognizing the voice, other may be good at recognizing them visually. The test on this showed varied results. But numerically it was bigger than the one-dimensional one.

This happens because we have different ways of storing information. For e.g: If someone speaks in other languages that we don't understand, we start representing them in English and our STM gives its own representation. So, its easier to hold the information in 3-D space. This is the primary reason why we can recognize a sentence, even if our 1-D recognition is very less than what we assume.

Since the memory span is a fixed number of chunks, we can increase the number of bits of information that it contains simply by building larger and larger chunks, each chunk containing more information than before.For e.g: We hear and then form words and then organize them into large chunks. This is something like representation 9800048486 as 98-000-48-48-6.

Paradox of Choice ( from speeches by Prof. Barry Schwartz ):

When we are exposed to a list of choices, we actually believe that we are having more freedom and hence more satisfied. But experimental results have proved that satisfaction decreases when the choices cross the numeric value (7±2 in most cases). Let me prove this from one of the experiments take from "Paradox of choice - Barry Schwartz". There are quite many but let me put down one.

Jam Example:

Experiment was conducted in Supermarket, as most of the decision making experiments are conducted in Super markets. For marketing the imported Jams, there were 26 choices of jams being placed as samples to be tasted. The offer was, if they buy the jam immediately after tasting, there was 1$ discount. This increased the crowd and some of them bought the Jam after tasting.

The same experiment was conducted after a few days, where only 6 samples were placed and the offer was the same. The crowd was less compared to 26 choices and some of them bought the Jam.

The result (not obvious) was that out of Jams sold from 6 samples, only 1/10 was sold with 26 choices.

There are many reasons why the increasing the choice reduced the sale. Some are:

• Regret of wrong choice: Most of the people are much worried about the wrong choice and since this feeling increases, they normally quit to take the choice.
• Opportunity cost: We make a choice, then we start thinking about the features that we are missing in others. This reduces our satisfaction and hence we quit.
• Increased Expectation: When the choices are more, expectations are more. And we start looking for the perfect ones.
• Self Blame: When the choice is less and we make a choice and if something is wrong, we blame the producers. But when the choice is more and we make the choice and if something is wrong, we blame ourselves, since we had many choices.
  

One solution for making decisions may be categorizing or reducing the chunks. If the 26 Jams are categorized into 6 categories, the choice becomes less and all the above factors come down. Or the second one is to make an agent select the choices or taking the opinion poll. Like we google for the best imported jam and then we find the agent making choice for us based on the opinion of others who have used. Here we are not making the choice, but we are depending on the agent to make the choice. So, the above mentioned factors won't appear.

"We might argue that in the course of evolution those organisms were most successful that were responsive to the widest range of stimulus energies in their environment. In order to survive in a constantly fluctuating world, it was better to have a little information about a lot of things than to have a lot of information about a small segment of the environment. If a compromise was necessary, the one we seem to have made is clearly the more adaptive."

- George Miller