From en.wikipedia.org/wiki/Grok

Grok is a verb roughly meaning "to understand completely" or more formally intuitive understanding. The term originated in Robert Heinlein's novel Stranger in a Strange Land, where it is part of the fictional Martian language and introduced to English speakers by a man raised by Martians.

In pursuit of knowledge...

Information is the mainstay of the 21st century. It is our universal currency and our staple nourishment.

When we talk about capital-I Information, what we are really talking about is heterogeneous information. And email re: Potluck dinner next Thursday. Blog posting about the quality of schools in Westchester county. Washington Post article about heart disease in men over 55. Business contact information for your real estate agent. Biographical data about Richard Strauss. Reminder to put the garbage out on Monday mornings. Itinerary for trip to Copenhagen. Child's report card. Book review of new Bob Dylan poetry reader. Price comparison for snazzy new PDA phones.

This makes sense, life is heterogeneous...and capital-I Information is only useful (usable) if you can effectively aggregate, process, manage and organize heterogeneous information in one place.

How do we effectively aggregate, process, manage and organize heterogeneous information in one place without being overwhelmed by the sheer volume and enormity of capital-I Information?

This is a loaded question and unfortunately requires a long-winded proposal as a response. However, the short quip reply is simply that whatever we do must do better than Search. Search, even in it's most exalted form, when it seemingly threatens to spillover into the realm of mind-reading and artifical intelligence is still a poor cousin to highly advanced reality-based information systems such as maps, wall calendars and whiteboards.

Search is like...murmuring a magic incantation, sticking your hand into an opaque bag and auto-magically having the very thing you wished for (or some close approximation) thrust into your hand by the (Google) genie inside the bag.

But you can't look inside the bag yourself. You don't even know how big the bag is. When you don't know what the right magic incantation is, you're done, game over. The genie inside the bag can't help you and you can't help yourself because you can't get inside the bag and "feel" your way to the right answer.

Search is not a solution to information management. It is emergency relief as we struggle to come up with a real answer.

Thus far, our real answers have proven to be less effective than our band-aid solution.

Folder-based hierarchies, Faceted systems, Tags, Mindmaps...Why is this? The short, flippant answer is: Because they all violate the fundamental Principles of Grok.

The magic number 7

People can hold 7 (+/- 2) units of information in short-term memory, and that's if they're trying.

This little nugget of wisdom has resulted in a lot of minimalist designs that are stingy with content to the point of being unusable.

• Hierarchies chunk down data by hiding data in opaque folders.
• Wizards chunk down data by hiding data behind opaque tabs.
• Microsoft Bob anyone?

People can hold 7 (+/- 2) units of information in short-term memory = People can only handle 7 (+/- 2) units of information.

If that were true, we would still be single-cell organisms living in a binary world of "salt, no salt" or "water, no water" or "sunlight, no sunlight".

A more nuanced statement might be: Any quantity of information is digestible so long as it is chunked down to 7 (+/- 2) chunks of information and within each chunk, the information is further chunked down to 7 (+/- 2) chunks of information and so on and so forth.

In fact, information is infinitely more digestible if the process of chunking it down doesn't end up hiding the information inside opaque containers (ie. folders or tabs in a wizard). Instead, as much information should remain exposed as possible in order to fully express the texture and nuance of what lies within the chunk.

Case study: NYT

There is a lot of information on the NYT homepage, but it's digestible because it is effectively chunked. The chunks are also prioritized by size and placement.

• NYT_analysis.png:
  

How much less comprehensible is the NYT homepage when the chunks are opaque?

• NYT_analysis_opaque.png:
  

Case study: DDC

Show the one where the most comprehensible parts of the DDC are the parts that have repetition. The parts that have repetition are the parts that have naturally chunked themselves down into digestible groupings.

• generalities.png:
  

Case study: File system hierarchy

In contrast, folder hierarchies are opaque in that the contents of folders are hidden from you so long as the folder remains closed. However, when the folders are actually opened the boundaries for each chunk of data (folder) become strangely hard to discern.

As a result there is no in-between state like the NYT and DDC examples above where you can see both clearly defined and transparent chunks of information.

• hierarchy.png:
  

A morsel for a monarch... is one that let's you Get and Forget

The process of "chunking" down data to the requisite 7 (+/- 2) morsels is complex and one filled with pitfalls. The making of a morsel is an art-form.

• A well-formed morsel can take on an infinite number of items and still be easily digested.
• An ill-formed morsel could have just 2 items in it and still be incomprehensible.

Ultimately, the true measure of worth for any morsel of information is how easy it is to Get and Forget the chunk.

A group of items constitute a grokkable chunk when the individual items corale around an indivisible concept or a spectrum concept.

An indivisible concept group usually means a relatively small set of data, less than 10 items. Chances are, if you have any more than 10 items, you have surpassed the limits of human memory and the chunk can probably be further sub-divided into smaller conceptual chunks.

A spectrum group is a set of items of uniform size that fill out a conceptual spectrum from end-to-end with no gaps and no areas of overlap. Spectrum groups can accomodate a potentially infinite set of items simply because they allow you to Get some governing principle for how items end up in the group and Forget about needing to parse the individual items in the group. By getting the underlying mechanism by which the group is formed, you can predict it's membership without actually having to examine the contents of the group.

Example of a mal-formed chunk:

• South America
• New York
• Starbucks on 81st and 2nd Avenue
• Yunnan Province
• 40�29'40"N to 45�0'42"N 71�47'25"W to 79�45'54"W
• My bedroom
• Pacific Northwest
• Conference room A/1742
• 231 Main Street
• Apt 4A
• The Bowery

Example of a well-formed indivisible concept group: Locations

• 231 Main Street
• 15 23rd Street
• 1 5th Avenue
• 82 5th Avenue
• 1992 Lafayette Street

Example of a well-formed spectrum concept group: Months of the Year

• January
• February
• March
• April
• May
• June
• July
• August
• September
• October
• November
• December

Example of a well-formed spectrum concept group: Integers: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50...

Example of a harder to grok spectrum concept group: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89...

The goal is to construct conceptual groupings that allow people to instantaneously understand the size, shape and nature of the data set either because it is so small that they can hold the entire data set in short-term memory or because it is such a homogeneous spectrum that someone could look at at just a handle of items in the set and predict the rest of the spectrum. Get and Forget.

A Well-formed spectrum concept group is filled out end-to-end and no areas of overlap

Filled out end-to-end means that there is never any doubt that you're missing out on some cache of misfiled information

• This week's status
  • Monday
  • Tuesday
  • Wednesday
  • Friday
• Where's Thursday?

People often deduce the right answer, by eliminating all of the obviously wrong answers thereby whittling the list of choices down to the one that sounds the least wrong (ie. SATs anyone). As a result, not being able to see all of the possible answers can sometimes be paralyzing.

No overlap diminishes confusion and ambiguity about where to look to find information.

Case study: Apple menus

The Mac's menu design illustrates the importance of filled-out spectrums to people's ability to deduce the right answer. Menu itemson Mac never change, even when they are meaningless in the current view. For example, the Mail.app Mailbox menu always offers both Go Online and Go Offline as options, even if the user is already Online or Offline.

The user might not know a priori that the Go Offline menu item will make the little whirly-gig animation stop going round and round without the aid of the greyed out Go Online menu item which tickles their memory and helps them remember how they got the app to "Start Syncing" in the first place.

• mailboxes_menu.png:
  

So what now?

What is needed is more than a better visual design for hierarchies. See Treemaps. One that is transparent with clearer boundaries for how the data is chunked. Looking at the folder hierarchy case study, the hierarchy would be hard to decipher even with the aid of the overlayed boxes. Looking at the NYT case study, it's clear that even a transparent hierarchical display of data with clearly define boundaries could handle no more than 2-3 levels of hierarchy at the most before distintegrating into undecipherable chaos.

This is because the levels of the hierarchy themselves become units of information and if you have 2-3 levels of hierarchy in each of the 4-5 sections of the NYT homepage, we're talking about 30-40 parent-child relationships to grok.

Now, if there was a clear and consistent rule governing the parent-child relationships, what's referred to as semantically pure hierarchy levels in the ... paper, then the 30-40 parent-child relationships could be quickly pared down to 2-3 general types of parent-child relationships.

Article-type>>Region>>Date.

The reality though, as discussed in the ... paper is that personal, user-defined hierarchies are never semantically pure and consistent and professionally constructed ones (ie. Dewey Decimal Classification system) aren't either.

But even IF a semantically pure and consistent hierarchy were possible first, to construct and second, to maintain...A deeply nested hierarchy would still quickly become ungrokkable simply because the number of relationships themselves would grow beyond the magic number: 7 (+/- 2) units of information.

Popularity>>Periodical>>Article-type>>Subject matter>>Sub-topic>>About whom>>Region>>State>>City>>Date>>Author>>

And this is because hierarchies have exactly 2 types of relationships: parent-child and sibling and doesn't allow for anything nuanced than that. First, not all relationships can be characterized as either parent-child or sibling. There are overlapping siblings. Overlapping facets. Threads...Hierarchies are unable to express these different types of relationship. And we are in turn, unable to further chunk down a deeply nested hierarchies by relationship type.

So what now?

As discussed in the ... paper, hierarchies are on some level a way to describe data, a way to encode both metadata and parent-child relationships between data.

• Your "To reply to" folder
• Your "Receipts" folder
• Your "Receipts" folder sub-divided by where the receipt is from: Amazon, eBay, Williams-Sonoma
• Your "East coast travel" folder sub-divided by state: Pennsylvania, Maine, South Carolina

But, not to beat a dead horse, hierarchies are more often than not, encoded inconsistently because they are more often than not, encoded in a bottom-up way on an as needed basis. In other words, any particular segment of the tree will make perfect sense. It is only when you try to zoom out a little bit and try to grok the tree as a whole that you get into all the various kinds of trouble described in graphic detail above.

As a result, we don't ever try to grok entire trees. Which brings us back to the third, underrated reason why people try to organize their information.

Instead, we search.

So, what now? See TheProblemOfTangibility