Recently in Types of Search Category

This week's thoughts come from the pile of serendipitous reading that routinely piles up on my desk. In this case a short article in Information Week caught my eye because it featured the husband of a former neighbor, Ken Krugler, co-founder of Krugle. I'd set it aside because a fellow, David Eddy, in my knowledge management forum group keeps telling us that we need tools to facilitate searching for old but still useful source code. In order to do it, he believes, we need an investment in semantic search tools that normalize the voluminous language variants scattered throughout source code. That would enable programmers to find code that could be re-purposed in new applications.

Now, I have taken the position that source code is just one set of intellectual property (IP) asset that is wasted, abandoned and warehoused for technology archaeologists of centuries hence. I just don't see a solid business case being made to develop search tools that will become a semantic search engine for proprietary treasure troves of code.

Enters old acquaintance Ken Krugler with what seems to be, at first glance, a Web search system that might be helpful for finding useful code out on the Web, including open source. I have finally visited his Web site and I see language and new offerings that intrigue me. "Krugle Enterprise is a valuable tool for anyone involved in software development. Krugle makes software development assets easily accessible and increases the value of a company's code base. By providing a normalized view into these assets, wherever they may be stored, Krugle delivers value to stakeholders throughout the enterprise." They could be onto something big. This is a kind of enterprise search I haven't really had time to think about but may-be I will now.

One thing leading to another, I checked out Ken Krugler's blog and saw an earlier posting: Is Writing Your Own Search Engine Hard? This is recommended reading for anyone who even dabbles in enterprise search technology but doesn't want to get her/his hands dirty with the mechanics. It is short, to-the-point and summarizes how and why so many variations of search are battling it out in the marketplace.

I don't want end-users to struggle too much with the under the hood details but when you are thinking about enterprise search for your organization, it is worth considering how much technology you are getting for the value you want it to deliver, year after year, as your mountains of IP content accrue. Don't give this idea short shrift because search is an investment that keeps giving if it is chosen appropriately for the problem you need to solve.

Called to account for the nomenclature “enterprise search,” which is my area of practice for The Gilbane Group, I will confess that the term has become as tiresome as any other category to which the marketplace gives full attention. But what is in a name, anyway? It is just a label and should not be expected to fully express every attribute it embodies. A year ago I defined it to mean any search done within the enterprise with a primary focus of internal content. “Enterprise” can be an entire organization, division, or group with a corpus of content it wants to have searched comprehensively with a single search engine.

A search engine does not need to be exclusive of all other search engines, nor must it be deployed to crawl and index every single repository in its path to be referred to as enterprise search. There are good and justifiable reasons to leave select repositories un-indexed that go beyond even security concerns, implied by the label “search behind the firewall.” I happen to believe that you can deploy enterprise search for enterprises that are quite open with their content and do not keep it behind a firewall (e.g. government agencies, or not-for-profits). You may also have enterprise search deployed with a set of content for the public you serve and for the internal audience. If the content being searched is substantively authored by the members of the organization or procured for their internal use, enterprise search engines are the appropriate class of products to consider. As you will learn from my forthcoming study, Enterprise Search Markets and Applications: Capitalizing on Emerging Demand, and that of Steve Arnold (Beyond Search) there are more than a lot of flavors out there, so you’ll need to move down the food chain of options to get it right for the application or problem you are trying to solve.

OK! Are you yet convinced that Microsoft is pitting itself squarely against Google? The Yahoo announcement of an offer to purchase for something north of $44 billion makes the previous acquisition of FAST for $1.2 billion pale. But I want to know how this squares with IBM, which has a partnership with Yahoo in the Yahoo edition of IBM’s OmniFind. This keeps the attorneys busy. Or may-be Microsoft will buy IBM, too.

Finally, this dog fight exposed in the Washington Post caught my eye, or did one of the dogs walk away with his tail between his legs? Google slams Autonomy – now, why would they do that?

I had other plans for this week’s blog but all the Patriots Super Bowl talk puts me in the mode for looking at other competitions. It is kind of fun.

That the Gilbane Group launched its Enterprise Search Practice this year was timely. In 2007 enterprise search become a distinct market force, capped off with Microsoft announcing in November that it has definitively joined the market.

Since Jan. 1, 2007, I have tried to bring attention to those issues that inform buyers and users about search technology. My intent has been to make it easier for those selecting a search tool while helping them to get a highly satisfactory result with minimal surprises. Playing coach and lead champion while clarifying options within enterprise search is a role I embrace. It is fitting then, that I wrap up this year with more insights gained from Gilbane Boston; these were not previously highlighted and relate to semantic search.

The semantic Web is a concept introduced almost ten years ago reflecting a vision of how the Worldwide Web (WWW) would evolve. In the beginning we needed a specific address (URL) to get to individual Web sites. Some of these had their own search engines while others were just pages of content we scrolled through or jumped through from link to link. Internet search engines like Alta Vista and Northern Light searched limited parts of the WWW. Then, Yahoo and Google came to provide much broader coverage of all "free" content. While popular search engines provided various categorizing, taxonomy navigation, keyword and advanced searching options, you had to know the terminology that content pages contained to find what you meant to retrieve. If your terms were not explicitly in the content, pages with synonymous or related meaning were not found. The semantic Web vision was to "understand" your inquiry intent and return meaningful results through its semantic algorithms.

The most recent Gilbane Boston conference featured presentations of commercial applications of various semantic search technologies that are contributing to enterprise search solutions. A few high level points gleaned from speakers on analytic and semantic technologies follow.

> Jordan Frank on blogs and wikis in enterprises articulated how they add context by tying content to people and other information like time. Human commentary is a significant content "contextualizer," my term, not his.
> Steve Cohen and Matt Kodama co-presented an application using technology (interpretive algorithms integrated with search) to elicit meaning from erratic and linguistically difficult (e.g. Arabic, Chinese) text in the global soup of content.
> Gary Carlson gave us understanding of how subject matter expertise contributes substantively to building terminology frameworks (aka "taxonomies") that are particularly meaningful within a unique knowledge community.
> Mike Moran helped us see how semantically improved search results can really improve the bottom line in the business sense in both his presentation and later in his blog, a follow-up to a question I posed during the session.
> Colin Britton described the value of semantic search to harvest and correlate data from highly disparate data sources needed to do criminal background checks.
> Kate Noerr explained the use of federating technologies to integrate search results in numerous scenarios, all significant and distinct ways to create semantic order (i.e. meaning) out of search results chaos.
> Bruce Molloy energized the late sessions with his description of how non-techies can create intelligent agents to find and feed colleagues relevant information by searching in the background in ways that go far beyond the typical keyword search.
> Finally, Sean Martin and John Stone co-presented an approach to computational data gathering and integrating the results in an analyzed and insightful format that reveals knowledge about the data, not previously understood.

Points taken are that each example represents a building block of the semantic retrieval framework we will encounter on the Web and within the enterprise. The semantic Web will not magically appear as a finished interface or product but it will become richer in how and what it helps us find. Similar evolutions will happen in the enterprise with a different focus, providing smarter paths for operating within business units.

There is much more to pass along in 2008 and I plan to continue with new topics relating to contextual analysis, the value, use and building of taxonomies, and the variety of applications of enterprise search tools. As for 2007, it's a wrap.

Following on my last post in which I covered the unique value propositions offered by a variety of enterprise search products, this one takes a look at the evolution of enterprise search. The commentary by search company experts, executives, and analysts indicates some evolutionary technologies and the escalation of certain themes in enterprise search. Furthermore, the pursuit of organizations to strengthen the link between searching technologies and knowledge enablers has never been more prominently featured taking search to a whole new level beyond mere retrieval.

The following paraphrased comments from the Enterprise Search Keynote session are timely and revealing. When I asked, Will Web and Internet Search Technologies Drive the Enterprise (Internal) Search Tool Offerings or Will the Markets Diverge?, these were some thoughts from the panelists.

Matt Brown, Principal Analyst from Forrester Research, commented that enterprise search demands much different and richer content interpretation types of search technologies. What Web-based searching does is create such high visibility for search that enterprises are being primed to adopt it, but only when it comes with enhanced capabilities.

Echoing Matt’s remarks, Oracle search solution manager Bob Bocchino commented on the difficulty of making search operate well within the enterprise because it needs to deal with structured database content and unstructured files, while also applying sophisticated security features that let only authorized viewers see restricted content. Furthermore, security must be deployed in a way that does not degrade performance while supporting continuous updates to content and permissions.

Hadley Reynolds, VP & Director of the Center for Search Innovation at Fast Search & Transfer, noted that the Web isn’t really making a direct impact on enterprise search innovation but many of the social tools found on the Web are being adopted in enterprises to create new kinds of content (e.g. social networks, blogs and wikis) with which enterprise search engines must cope in richer contextual ways.

Don Dodge, Director of Business Development for the Emerging Business Team at Microsoft further noted that the Internet’s biggest problem is scale. That is a much easier problem to solve than in the enterprise where user standards for what qualifies as a good and valuable search results are much higher, therefore making the technology to deliver those results more difficult.

Among the other noteworthy comments in this session was a negative about taxonomies. The gist of it was that they require so much discipline that they might work for a while but can’t really be sustained. If this attitude becomes the norm, many of the semantic search engines which depend on some type of classification and categorization according to industry terminologies or locally maintained lists will be challenged to deliver enhanced search results. This is a subject to be taken up in a later blog entry.

A final conclusion about enterprise search was a remark about the evolution of adoption in the marketplace. Simply put, the marketplace is not monolithic in its requirements. The diversity of demands on search technologies has been a disincentive for vendors to focus on distinct niches and place more effort on areas like e-commerce. This seems to be shifting, especially with all the large software companies now seriously announcing products in the enterprise search market.

In my quest to analyze the search tools that are available to the enterprise, I spend a lot of time searching. These searches use conventional on-line search tools, and my own database of citations that link to articles, long forgotten. But true insights about products and markets usually come through the old-fashioned route, the serendipity of routine life. For me search also includes the ordinary things I do everyday:
> Looking up a fact (e.g. phone number, someone’s birthday, woodchuck deterrents), which I may find in an electronic file or hardcopy
> Retrieving a specific document (e.g. an expense form, policy statement, or ISO standard), which may be on-line or in my file cabinet
> Finding evidence (e.g. examining search logs to understand how people are using a search engine, looking for a woodchuck hole near my garden, examining my tires for uneven tread wear), which requires viewing electronic files or my physical environment
> Discovering who the experts are on a topic or what expertise my associates have (e.g. looking up topics to see who has written or spoken, reading resumes or biographies to uncover experience), which is more often done on-line but may be buried in a 20-year old professional directory on the shelf
> Learning about a subject I want or need to understand (e.g. How are search and text analytics being used together in business enterprises? what is the meaning of the tag line “Turbo Search Engine” on an Acura ad?), which were partially answered with online search but also by attending conferences like the Text Analytics Summit 2007 this week

This list illustrates several things. First search is about finding facts, evidence, aggregated information (documents). It is also about discovering, learning and uncovering information that we can then analyze for any number of decisions or potential actions.

Second, search enables us to function more efficiently in all of our worldly activities, execute our jobs, increase our own expertise and generally feed our brains.

Third, search does not require the use of electronic technology, nor sophisticated tools, just our amazing senses: sight, hearing, touch, smell and taste.

Fourth, that what Google now defines as “cloud computing” and MIT geeks began touting as “wearable” technology a few years ago have converged to bring us cars embedded with what Acura defines as “turbo search engines.” On this fourth point, I needed to discover the point. In small print on the full page ad in Newsweek were phrases like “linked to over 7,000,000 destinations” and “knows where traffic is.” In even tinier print was the statement, “real-time traffic monitoring available in select markets…” I thought I understood that they were promoting the pervasiveness of search potential through the car’s extensive technological features. Then I searched the Internet for the phrase “turbo search engine” coupled with “Acura” only to learn that there was more to it. Notably, there is the “…image-tagging campaign that enables the targeted audience to use their fully-integrated mobile devices to be part of the promotion.” You can read the context yourself.

Well, I am still trying to get my head around this fourth point to understand how important it is to helping companies find solid, practical search solutions to problems they face in business enterprises. I don’t believe that a parking lot full of Acura’s is something I will recommend.

Fifth, I experienced some additional thoughts about the place for search technology this week. Technology experts like Sue Feldman of IDC and Fran Halper of Hurwitz & Associates appeared on a panel at the Text Analytics Summit. While making clear the distinctions between search and text analytics, and text analytics and text mining, Sue also made clear that algorithmic techniques employed by the various tools being demonstrated are distinct for each solving different problems in different business situations. She and others acknowledge that finally, having embraced search, enterprises are now adopting significant applications using text analytic techniques to make better sense of all the found content.

Integration was a recurring theme at the conference, even as it was also obvious that no one product embodies the full range of text search, mining and analytics that any one enterprise might need. When tools and technologies are procured in silos, good integration is a tough proposition, and a costly one. Tacking on one product after another and trying to retrofit to provide a seamless continuum from capturing, storing, and organizing content to retrieving and analyzing the text in it, takes forethought and intelligent human design. Even if you can’t procure the whole solution to all your problems at once, and who can, you do need a vision of where you are going to end up so that each deployment is a building block to the whole architecture.

There is a lot to discover at conferences that can’t be learned through search, like what you absorb in a random mix of presentations, discussions and demos that can lead to new insights or just a confirmation of the optimal path to a cohesive plan.

My silence last week had more to do with information overload than lack of interesting things to write about. Be forewarned, the floodgates of my brain are beginning to creak open. I just returned from Fast Search’s FastForward 07 conference in San Diego where their current and future visions for search technologies were front and center. While there seem to be no lack of innovations for how to make search engines smarter, faster, and more adaptable, the innovations being hyped at FastForward 07, and by others with only slightly less hyperbole, are notable. Search is becoming sexy and not just for the amount of money that Google and second-ranked Fast are raking in. In this arena search is the new business frontier, the marketplace-enabler, the marketplace-maker.

Consider this, search technologies have been business necessities for 35 years. For the first 30, search was strictly a support feature to many other kinds of finding mechanisms. In the earliest days search was performed by specialists as a service to other operations in the organization. Attempts to market search technology options to line managers, analysts, attorneys and R&D staff were marginal in their success. This is because search was not used enough for these groups to acquire the skill required for it to be really valuable. Once Web search engines exposed everyone to the possibilities of search in a far simpler modality, the innovation light bulbs popped off.

Suddenly search for use within the enterprise has become search for the enterprise’s marketplace, a major business driver that will put an organization’s products, services, and assets squarely in front of the right buying audience. What this means for those poor souls who still need to find the stuff mounting valuelessly in inaccessible silos remains to be seen. I am excited by what I saw but concerned by what I am witnessing. It is great that I may be able to find that weird audio adapter on the Web to let me connect to the sound system in the skating rink. But it is really awful when an engineering firm can’t put it’s hands on the schematic that shows how a circuit board was modified and delivered three years ago to a top customer.

I seem to be attending a lot of presentations because I think they are going to be about “enterprise search.” Instead they cover a new offering or positioning strategy by a search company seeking to help enterprises monetize their Web sites. There are great business models in this space as Yahoo, Google and Amazon have illustrated. These will morph to offerings, as yet, unimagined. The trouble is that for my audience, I want to help them understand offerings that will help them with searching content already in the enterprise or from outside that they can leverage for business uses: competitive intelligence, product development, supply chain improvements, marketing collateral development. That is what enterprise search is to most people working inside organizations. Admittedly, this is not a new or “sexy” market. I think that vendors of search may be so worn down by how they can make money offering their search tools for searching inside the organization that they may just be talking up other markets to stave off their own boredom with the “inside the enterprise market.” Horizontal markets are tough to deal with (more about that in a later blog entry).

To all you vendors who would like to cut and run, I respectfully ask that you stay. There is a crying need albeit a very big financial challenge in all of this. Enterprises have no idea what their true monetary loses are because workers can’t find “stuff.” There have been plenty of guesses put forth by analysts, but until we see search solutions that don’t take decades (OK years) to implement, who actually knows what gains could be made by really good and easy search tools that find both structured and unstructured content with a minimum of set-up. The pricing models of tools need to make better sense, including ways to chunk the expenditures incrementally with “quick start,” and low overhead options. Why do the search tools with the mightiest claims also come with the highest price tags for licensing but the least out-of-the box functionality?

To all you enterprise information technology seekers of “true enterprise search” you bear responsibility for some of the mess this market is in. You’ve got to write better specifications, learn to start small and demand small to get started, and come to the selection process with real users and professional searchers on the team who will test drive products before they are purchased. If a product doesn’t solve the specific search problem you are trying to solve, don’t buy it. May-be it doesn’t feel like your money when you purchase for your enterprise but it really is your professional responsibility. Would you buy a car that will only take you to the beach or a lake but not to the grocery store?

Enterprise Search has been an illusive dream for too many organizations for too many years. Search technology is ubiquitous but the “holy grail” for most organizations is to be able to find all content within the organization through a single query interface. My instinct is to give a chronology of search over the past four or five decades to guide your understanding of why enterprise search has remained so “out of reach.” I could also describe the ways in which search technologies have evolved and morphed with hundreds of functions and thousands of features. It would certainly help explain why the typical company has a daunting task narrowing its options but it would probably not quicken the selection process.

For now, one view of the current market segmentation is a starting point. Sue Feldman, Research VP, Content Management and Retrieval Solutions at IDC, gave the audience a high level view of the market in a session at Gilbane Boston 2006. She placed enterprise search technology into three big buckets: Appliances and Downloadable Search, Enterprise Search (software) Platforms, and Application Specific Search embedded with other software. She then broadly described the features and functions that characterize each major type. If you have grown up with search in your professional life for over 30 years as I have, it makes perfect sense that this is what we have come to in the market but differentiating the options is a step far less clear-cut.

After the sessions, 15 conference-goers joined me to continue discussing and learning about enterprise search in a roundtable forum. It was hard to know which end of the search animal we should address first to help everyone speak the same language. That is precisely what is making this marketplace such a tough one. Vendors represent a huge variety of solutions, each positioning product(s) for a problem of their definition, offering technology that targets the specific problem. Buyers have multiple search needs but still want a single solution. Further complicating the mix is a dizzying array of search jargon. With vendors and buyers using their own language the market is, frankly, a real mess.

Take Ms. Feldman’s three big buckets and think of one example of search product in each category. Now think about all the types of searches that people in your organization need to perform just to get their routine work done:
> Looking up an address in a directory
> Finding an image for a presentation
> Retrieving a press release your department issued last year on a new product
> Locating a configuration change to a piece of equipment in manufacturing
> and so on…

Can you imagine any single search interface or product from the tools you know that would give you the means to find all of these pieces of information? Can you imagine a single search tool that would answer your query in a couple of simple steps, and able to perform the functions right out of the box? Simple solutions that address the complexity of business variables and technology standards in most organizations make any single solution an unlikely candidate at a reasonable cost.

Blog readers can request answers to questions, ask for help with sorting out the marketplace or definitions to understand the jargon. I invite readers to tell me what you think needs to be talked about and I’ll give it my best shot. What do you need to know first to tread through the search marketplace?