No

Search hype

For the last few years, search (by search I mean both traditional notions of search and tag and label-based "human-enhanced" search systems* ) has increasingly become the primary and for some, sole paradigm in which people interact with information: private, shared and public.

Web search: Google, MSN, Yahoo
Desktop search: Google, Yahoo, MSN, Spotlight, Quicksilver
Search-based email: gMail, Bloomba, OperaMail^?
Tag or Label-based data systems: Delicious, Flickr, gMail

Computers freed us from our filing cabinets. Google freed us from our digital folders.

It's as if someone has told us that collectively as a society, we would never have to fold a single item of laundry ever again. Which is indeed, cause to celebrate. But what are we giving up by not folding our laundry? Mismatched socks? Wrinkled clothes? Missed opportunity?

In our euphoria over the success and surprising usefulness and usable-ness of search and tagging, are we passing by a chance to not only make information accessible and useful but fundamentally alter the way we use information to make decisions.

A brief history of human interaction with information

The first written languages fundamentally changed the way people absorbed and managed information.

For the first time, information could be put down and recorded in an objective medium, removed from the subjective data store known as human memory.

Oral and Aural traditions became Written traditions that persisted through time, unblemished by personal embellishments and plain forgetfulness. (Stories also got less repetitive and didn't always have to rhyme anymore.)

Although I do not know enough about neurophysiology to speak definitely on this Information transfer itself probably transferred to different parts of the brain (Oral and Aural versus Conceptual and Visual).

Since that disruptive, course-changing period in history, subsequent innovations in information technology: papyrus, ink, fountain pens, printing press, typewriters, ball-point pens, word processors, computers, the internet, have been elaborations on the original innovation. Useful and earth-moving elaborations: Making information transfer easier, faster, more widely disseminated, more easily duplicated. But on some theoretical level, they remain variations on a theme.

...Because we have yet to make a fundamental shift in the "cognitive" laws of physics of how we process and manage information.

This is an overblown way of saying, we did not come all this way to end in Search, the biggest leap in information technology is yet to come.

If the invention of written language extracted data out of the real world by encoding and "virtualizing" it into an alphabet or a set of ideograms...Then the obvious next step in the spiral progression of history is to decode that code and re-present data in a way that is more true to how it exists in the "real world" without losing the value-add of virtualized, encoded data: portability, malleability, persistence through time and the ability to be anywhere and everywhere.

The next big problem to solve

Search doesn't solve many of the biggest problems we face in our still nascent yet rapidly growing information-centric, information overloaded age. Specifically: how do we grok, wrap our heads around all this data? (NOT just succeed in finding stuff in it.)

But we can't solve these problems until we get past Step 1: Admit we have a problem...

The reason we're having such a hard time getting a handle on all this information is ironically precisely because we're not actually able to get the metadata that will help us tie together and understand the heterogeneous mess of information we deal with everyday.

In some sense, we're not overloaded with too much information. While we get a lot of data electronically, we don't really do very much with it beyond remember that we have it, look it up and look at it. We store it in various places and in various forms that have little or nothing to do with the information itself: Documents as attachments in email clients, URLs in spreadsheets, anything and everything in folders. There is a lot of information we could get out of the data we have that we're simply not aware of or don't have ready access to it.

• Show me all of the apartments I thought were acceptable in my search for a place to live. How would I stack rank them relative to one another? Does my ranking correspond to price? What's my price range given my standards for quality of life and market rates for the apartments I deemed acceptable? Do I need to lower my standards or raise my price range?
• How many versions of this draft have we passed back and forth? How long have we been working on it? How many times have we revised it? What were the most significant changes? Are there other documents we could draw material from?
• What are all of the vacation destinations I've researched? Are there any places that are especially good to visit in the next month? Because it's the off-season. Because airfare is cheap. Because the weather is nice. Because it's Oktoberfest.
• What are all of the things I need to follow-up on? Are any of them dependent on other people? Are other people depending on me? Who are these people? How important are they to me? Am I prioritizing correctly? Are any of the tasks dependent on other tasks? Are there tasks that have a lot of tasks depending on it?
• How much money am I spending? Where can I save the most money with the least amount of effort? How much money do I spend on others versus on myself?

It is this metadata: data about our data that is the common glue tying together your disparate, heterogeneous data. URLs, notes, documents, emails, calendar events, spreadsheets, music, etc.

This data on your computer can't offer you a complete picture of your life, but it can at least provide you with a picture of the part of your life that leaves a digital imprint (which in modern-day, first-world life is a signficant and ever-expanding part of the pie).

Why Search doesn't solve The Next Big Problem

Search is like murmuring a magic incantation, sticking your hand into an opaque bag and auto-magically having the very thing you wished for (or some fistful of things, one of which is the very thing you wished for) thrust into your hand by the (Google) genie inside the bag.

But you can't look inside the bag yourself. You don't even know how big the bag is. When you don't know what the right magic incantation is, you're done, game over. The genie inside the bag can't help you and you can't help yourself because you can't get inside the bag and "navigate" your way to the right answer.

Search is not a solution to information management. It's one piece of the puzzle.

The crux of the problem is that: Search doesn't offer you information, you didn't already know, it's value proposition is in locating something you already know about.

Some might argue, this is good enough. It solves for 80% of the information overload problem. But what if all the interesting things to come are hiding in that last 20%?

This is a morbid example, but I recently ran across an example that struck a bell. James Baldwin, in his account of a serial murder case involving 28 young black boys in Atlanta The Evidence Of Things Not Seen describes a police river dredging effort to look for the bodies of 2 missing boys believed to have fallen victim to the serial killer. They fail in their search for the boys, but in the process, dredge up almost a dozen unidentified bodies. More bodies than there were open murder cases at the time.

Given a tag and search based system, searching on the term missing person would yield and overwhelming list of results. Searching on a missing person would yield only the information you already know about that missing person. How do you find the information that "might" be related to the missing person you are trying to find, but is several degrees removed from that person?

So if I were looking for John Doe, and I had a tag: John Doe, which intersected with the tag: missing persons, which in turn intersected with the tag: serial murder investigation. The piece of crucial information falls in the intersection of tag: missing persons and tag: serial murder investigation. But, given that searching on missing persons yields such an overwhelming list of results: Would it have occured to me to narrow down that list of results by either searching for "missing persons" and "serial murder investigation", if I John Doe had been missing for decades and had absolutely nothing to do with the current murder case?

• In Search of John Doe:
  

But the problem is greater than simply not being able to find what you want...

Gestalt and the eternal search for a coherent synthesis of information is fundamental to how humans take in, process and come to understand information. And there is strong argument in favor of the idea that unless you have a top-down understanding of something, you don't really understand it at all.

Deriving a top-down view from bottom-up experiences

It's true that over time, as you become more familiar with a data set, the top-down mental picture can grow out of your bottom-up interactions with the data. This accounts for why people are so much more at ease navigating within their personal data structures than in public ones or someone else's.

But I would argue that

1 Top-down mental maps derived from bottom-up interactions are unreliable and subjective. (Not to mention, hard to maintain. (Have you ever had to consult a map to settle a disagreement with a back seat driver as to directions, even in a neighborhood you're both fairly familiar with?)

Even with relatively static, homogeneous sets of data (my iTunes repository), I find myself consistently missing songs because I don't know how to find them.

2 Static and homogeneous is exactly the opposite of what real-world information is like: heterogeneous, unpredictable, rapidly and constantly growing. In other words, the data can't wait around for us to slowly formulate our top-down conceptions over time. It has changed and rendered our mental maps out-dated and obsolete before we've had a chance to build them.

Fortunately, the very attributes that make information technology overwhelming and disorienting (ie. it's intangible, virtual nature which makes it disconcertingly easy to mutate, move around and multiply rapidly and indiscriminately) are also the very characteristics that pave the way for the next leap in how humans will interact with information.

The rest of this paper will be an exploration of how data management systems need to go beyond simply being effective data stores with powerful targeted retrieval (search) tools to systems that pull together synthesized, top-down views of our ballooning, changing and heterogeneous data.

This is cognitive leap we're about to make with information technology, the ability to transcend time with a system that aggregates and synthesizes data in ways that could only be done "in your head" before.

• What are the consequences of this?
• How will this change our ability to make decisions and set priorities?
• How does this change an individual's ability to communicate and share information with others?

The questions remain: How do we accomplish such a thing? Is it even possible?

*Tagging as a form of human-enhanced search

If you think about the tagging experience, it's really very similar to the search experience. You don't really start out top-down the way you do in hierarchies, surveying the landscape of browsing possibilities and then narrowing in on a particular branch or cluster of tags. There are simply too many possibilities, too many tags.

Tagging is simply humans helping computers. Instead of relying on the search engine to figure out what an item is about and how it might be significant to you, you explicitly extrapolate that meta-data from the guts of the information item and hang it off the Tag: attribute of the item as a handle for the search engine.

Fundamentally, whether you help the search engine or not, the experience is the same. You're a blind mole rat navigating an unfamiliar maze of underground tunnels, hopping from tag to tag without any clear picture of how it all fits together. You are not a hawk surveying the lay of the land looking for a tasty morsel and finding it in the unlikeliest of places.