Enterprise Search Practice Blog: Individual Author Entries

Search your Enterprise Really Needs

In the forthcoming Gilbane research report, Enterprise Search Markets and Applications Capitalizing on Emerging Demand, I describe several distinctly different scenarios for search applications. The variety of search products underscores innovative approaches to applying search and diversity of needs. The Enterprise Search track at the upcoming Gilbane Conference in San Francisco will feature numerous examples of why and how search is being applied across small, medium and very large domains of enterprise content. Hearing from those experienced in implementing and deploying search solutions will inform you when positioning your search "must haves" as you narrow possible options.

Our first group panel will feature two consultants and a solutions provider each with a perspective on aligning the search problem you are trying to solve with a business case and the type of product being offered. As moderator, I will be looking for examples from speakers that will resonate with the audience to provide a connection between what has been demonstrated as valuable and workable, and what conference-goers are seeking. These sessions are about matching experience with investigation and creating an environment for exchanging information and allowing inquiry and research to flourish.

Much has been made of the rise of "social" technologies in the past year, but technology is only a tool. Any meeting gathering with product exhibits facilitates your first-hand viewing of technology and the vendors offering products. But more important, are the professional social connections that give flesh and realism to the application of technology. If you set out to ask just one question of each speaker you meet or fellow attendee, make it one that will help you build a realistic picture around a product you are considering to meet a need. For example, ask not about whether product "A" can perform function "XYZ" but what it took in terms of human resources to deliver that terrific interface that the speaker is showcasing. Social networking gives you that opportunity.

Professional conferences are learning opportunities and, compared to today's college tuition costs, a great bargain. Also, educational institutions are relatively limited, exposing you to controlled scenarios or short-term experiences. What you gain at meetings like the Gilbane conferences is opportunity to benefit from long-term experiences in real business situations by asking those who have been there and done that, how it came about, got built and what the demonstrable outcomes are.

A look at these topics for session EST-2 shows how our speakers will frame their experiences: Venkat Rangan, CTO, Clearwell Systems, Search and Information Retrieval Needs for eDiscovery; Randy Woods, Executive VP, Non-linear Creations, Best Practices for Tuning Enterprise Search; Sam Mefford, Enterprise Search Practice Lead, Avalon Consulting, Beyond Silos: Changing 'Hide and Seek' to 'Index and Find.' I'm always looking for new perspectives on search and ways of helping my clients understand their options. This will enrich my own learning experiences, as well.

Interview with Steve Arnold about "Beyond Search" with Jess Bratcher

Interview with Arnold-04162008.pdf

Enterprise Search and the Conference Season

My blog has been silent for several weeks as I wrapped up a study of the enterprise search marketplace. More information about the report will be forthcoming in the next week or so. In the meantime, the conference season is upon us with the Infonortics Search Engine conference just held in Boston, the Enterprise Search Summit in New York next week, TextAnalytics being held in Boston in mid-June and our own Gilbane San Francisco Conference being held June 18 - 20th. It is a feast for those in the market to buy or just become more familiar with the huge number of options. In my recent research on the marketplace I interviewed a number of people who had recently made a procurement of a search product. To a person there was significant pain expressed about how much time had been spent examining and rejecting options. With well over 100 search and "beyond search" products that are now commercially viable on the market, you need to find ways to winnow your choices efficiently. There is no better way to do this than to acquire publications that give you comprehensive information concentrated in one place PLUS going to conferences to:
a) To meet vendors and assess the type of business relationship you are likely to experience with them
b) Meet other users or potential users of the various technologies to learn, first hand, what their experiences have been buying and using search software

Attending conference sessions where case studies are being given by those deploying or using software is important, but discussions on the side can also be valuable. People who show up at our Gilbane Conferences are a sharing crowd and are easy to network with. As the track chairman for all the enterprise search sessions in San Francisco, I plan to hold at least one and maybe two roundtable discussions, open to anyone who wants to participate in a free flow of ideas about enterprise search. This will likely be in the location of the lunch venue - so we can pick at our food and each others' brains, simultaneously.

Over the next couple of weeks, I plan to showcase the themes for our search sessions in San Francisco, beginning with the Search Keynote. Last year in Boston we had a panel discussion of search executives and analysts; that was a great discussion. This June I am going to thrust Steve Arnold, author of our new publication Beyond Search, into our spotlight with a series of questions about the marketplace to discover things that he thinks buyers should be focused on over the next six months plus soliciting some thoughts on selecting appropriate technologies. He will surely add commentary on the changing vendor landscape and what it means. Once I have had a go at questioning him, the audience will have a chance to seek his guidance. This is a "not to be missed" session so please put it on your calendar - it will not be recorded.

To warm you up to Mr. Arnold's style and range of thoughts on the subject, check out this recent interview he gave to Jess Bratcher of Bratcher & Associates.

Only Humans can Ensure the Value of Search in Your Enterprise

While considering what is most important in selecting the search tools for any given enterprise application, I took a few minutes off to look at the New York Times. This article, He Wrote 200,000 Books (but Computers Did Some of the Work), by Noam Cohen, gave me an idea about how to compare Internet search with enterprise search.

A staple of librarians' reference and research arsenal has been a category of reference material called "bibliographies of bibliographies." These works, specific to a subject domain, are aimed at a usually scholarly audience to bring a vast amount of content into focus for the researcher. Judging from the article, that is what Mr. Parker's artificial intelligence is doing for the average person who needs general information about a topic. According to at least one reader, the results are hardly scholarly.

This article points out several things about computerized searching:

It does a very good job of finding a lot of information easily.

Generalized Internet searching retrieves only publicly accessible, free-for-consumption, content.

Publicly available content is not universally vetted for accuracy, authoritativeness, trustworthiness, or comprehensiveness, even though it may be all of these things.

Vast amounts of accurate, authoritative, trustworthy and comprehensive content does exist in electronic formats that search algorithms used by Mr. Parker or the rest of us on the Internet will never see. That is because it is behind-the-firewall or accessible only through permission (e.g. subscription, need-to-know). None of his published books will serve up that content.

Another concept that librarians and scholars understand is that of primary source material. It is original content, developed (written, recorded) by human beings as a result of thought, new analysis of existing content, bench science, or engineering. It is often judged, vetted, approved or otherwise deemed worthy of the primary source label by peers in the workplace, professional societies or professional publishers of scholarly journals. It is often the substance of what get republished as secondary and tertiary sources (e.g. review articles, bibliographies, books).

We all need secondary and tertiary sources to do our work, learn new things, and understand our work and our world better. However, advances in technology, business operations, and innovation depend on sharing primary source material in thoughtfully constructed domains in our enterprises of business, healthcare, or non-profits. Patient's laboratory or mechanical device test data that spark creation of primary source content need surrounding context to be properly understood and assessed for value and relevancy.

To be valuable enterprise search needs to deliver context, relevance, opportunities for analysis and evaluation, and retrieval modes that give the best results for any user seeking valid content. There is a lot that computerized enterprise search can do to facilitate this type of research but that is not the whole story. There must still be real people who select the most appropriate search product for that enterprise and that defined business case. They must also decide content to be indexed by the search engine based on its value, what can be secured with proper authentication, how it should be categorized appropriately, and so on. To throw a computer search application at any retrieval need without human oversight is a waste of capital. It will result in disappointment, cynicism and skepticism about the value of automating search because the resulting output will be no better than Mr. Parker's books.

Parsing the Enterprise Search Landscape

Steve Arnold's Beyond Search report is finally launched and ready for purchase. Reviewing it gave me a different perspective on how to look at the array of 83 search companies I am juggling in my upcoming report: Enterprise Search Markets and Applications. For example, technological differentiators can channel your decisions about must haves/have nots in your system selection. Steve codifies considerations and details 15 technology tips that will help you frame those considerations.

We are getting ready for the third Gilbane Conference in which "search" has been a significant part of the presentation landscape in San Francisco, June 17 - 20th.Six sessions will be filled with case studies and enlightening "how-to-do-it-better" guidance from search experts with significant "hands-on" experience in the field. I will be conducting a workshop, immediately after the conference, How to Successfully Adopt and Deploy Search. Presentations by speakers and the workshop will focus on users' experiences and guidance for evaluating, buying and implementing search. Viewing search from a usage perspective begs a different set of classification criteria for divvying up the products.

In February, Business Trends published an interview I gave them in December, Revving up Search Engines in the Enterprise. There probably isn't much new in it for those who routinely follow this topic but if you are trying to find ways to explain what it is, why and how to get started, you might find some ideas for opening the discussion with others in your business setting. The intended audience is those who don't normally wallow in search jargon. This interview pretty much covers the what, why, who, and when to jump into procuring search tools for the enterprise.

For my report, I have been very pleased with discussions I've had with a couple dozen people immersed in evaluating and implementing search for their organizations. Hearing them describe their experiences guides other ways to organize a potpourri of search products and how buyers should approach their selection. With over eighty products we have a challenge in how to parse the domain. I am segmenting the market space into multiple dimensions from the content type being targeted by "search" to the packaging models the vendors offer. When laying out a simple "ontology" of concepts surrounding the search product domain, I hope to clarify why there are so many ways of grouping the tools and products being offered. If vendors read the report to decide which buckets they belong in for marketing and buyers are able to sort out the type of product they need, the report will have achieved one positive outcome. In the meantime, read Frank Gilbane's take on the whole topic of enterprise tacked onto any group of products.

As serendipity would have it, a colleague from Boston KM Forum, Marc Solomon, just wrote a blog on a new way of thinking of the business of classifying anything, "Word Algebra." And guess who gave him the inspiration, Mr. Search himself, Steve Arnold. As a former indexer and taxonomist I appreciate this positioning of applied classification. Thinking about why we search gives us a good idea for how to parse content for consumption. Our parameters for search selection must be driven by that WHY?

Enterprise Search Adopters

May-be it is this everlasting winter of weather events, but I'm ready for some big changes across the gray landscape. Experiencing endless winter has for me become a metaphor for what I observe within some enterprises as serial adoptions of search.

As I work on my forthcoming report, Enterprise Search Markets and Applications: Capitalizing on Emerging Demand, I am interviewing people who are deeply engaged in search technologies. They are presenting a view of search deployment and implementation that reinforces my own observations, complete with benefits and disappointments. However, search in enterprises is like recurring weather events, some big, some small but relentless in the repetitiveness of certain experiences. It seems that early adopters in the early stages of adoption often experience the euphoria of a fresh way to find stuff. Then inertia sets in as some large subset of adopters settles in to becoming routine but faithful users. The rest are like me with winter, looking for a really big change and more; the nitpicking begins as users cast their eyes to better options hyped by the media or by compatriots in other organizations with newer "bells and whistles." Ah, what fickle beasts we are, as my husband will be very quick to remind me the first hot, humid day of summer when I complain in a desultory sulk.

So, I was delighted to read this article in the New York Times, Tech's Late Adopters Prefer the Tried and True, by Miguel Helft, on March 12. I particularly loved this comment from the article: "Laggards have a bad rap, but they are crucial in pacing the nature of change, said Paul Saffo, a technology forecaster in Silicon Valley. Innovation requires the push of early adopters and the pull of laypeople asking whether something really works. If this was a world in which only early adopters got to choose, we'd all be using CB radios and quadraphonic stereo." It helps to put one's quest for the next big thing into perspective.

It included another quote from David Gans who, from the community of the Well in which people communicate using text-only systems, "Just because you have a nuclear-powered thing that can dry your clothes in five minutes doesn't mean there isn't value to hanging your clothes in the backyard and talking to your neighbors while doing it." As one who has never owned a clothes drier, this validated one of my own conscious decisions.

Seriously though, given all the comments collected from my interviews and my own experiences, it is really time to remind adopters, early and late, to give thought to appropriateness, what benefits us or adversely distracts us in the technologies we implement in our working worlds. (I'll leave your personal technology use for you to sort out.) Taking time to think about your intentions and "what comes next" after getting that "must have" new search system is something only you can control. Nobody on the selling side of a bakery will ever remind you that you don't really neeeed another cookie.

And in one more point, if you are in the market for search+, Steve Arnold does a fine job of positioning the appropriateness of each of the 24 systems he reviews in Beyond Search. It might just help you resist the superfluous and take a look some other options instead.

Ontologies and Semantic Search

Recent studies describe the negative effect of media including video, television and on-line content on attention spans and even comprehension. One such study suggests that the piling on of content accrued from multiple sources throughout our work and leisure hours has saturated us to the point of making us information filterers more than information "comprehenders". Hold that thought while I present a second one.

Last week's blog entry reflected on intellectual property (IP) and knowledge assets and the value of taxonomies as aids to organizing and finding these valued resources. The idea of making search engines better or more precise in finding relevant content is edging into our enterprises through semantic technologies. These are search tools that are better at finding concepts, synonymous terms, and similar or related topics when we execute a search. You'll find an in depth discussion of some of these in the forthcoming publication, Beyond Search by Steve Arnold. However, semantic search requires more sophisticated concept maps than taxonomy. It requires ontology, rich representations of a web of concepts complete with all types of term relationships.

My first comment about a trend toward just browsing and filtering content for relevance to our work, and the second one about the idea of assembling semantically relevant content for better search precision are two sides of a business problem that hundreds of entrepreneurs are grappling with, semantic technologies.

Two weeks ago, I helped to moderate a meeting on the subject, entitled Semantic Web - Ripe for Commercialization? While the assumed audience was to be a broad business group of VCs, financiers, legal and business management professionals, it turned out to have a lot of technology types. They had some pretty heavy questions and comments about how search engines handle inference and its methods for extracting meaning from content. Semantic search engines need to understand both the query and the target content to retrieve contextually relevant content.

Keynote speakers and some of the panelists introduced the concept of ontologies as being an essential backbone to semantic search. From that came a lot of discussion about how and where these ontologies originate, how and who vets them for authoritativeness, and how their development in under-funded subject areas will occur. There were no clear answers.

Here I want to give a quick definition for ontology. It is a concept map of terminology which, when richly populated, reflects all the possible semantic relationships that might be inferred from different ways that terms are assembled in human language. A subject specific ontology is more easily understood in a graphical representation. Ontologies also help to inform semantic search engines by contributing to an automated deconstruction of a query (making sense out of what the searcher wants to know) and automated deconstruction of the content to be indexed and searched. Good semantic search, therefore, depends on excellent ontologies.

To see a very simple example of an ontology related to "roadway", check out this image. Keep in mind that before you aspire to implementing a semantic search engine in your enterprise, you want to be sure that there is a trusted ontology somewhere in the mix of tools to help the search engine retrieve results relevant to your unique audience.

Taxonomy and Enterprise Search

This blog entry on the "Taxonomy Watch" website prompts me to correct the impression that I believe naysayers who say that taxonomies take too much time and effort to be valuable. Nothing could be further from the truth. I believe in and have always been highly vested in taxonomies because I am convinced that an investment in pre-processing enterprise generated content into meaningfully organized results brings large returns in time savings for a searcher. S/he, otherwise, needs to invest personally in the laborious post-processing activity of sifting and rejecting piles of non-relevant content. Consider that categorizing content well and only once brings benefit repeatedly to all who search an enterprise corpus.

Prime assets of enterprises are people and their knowledge; the resulting captured information can be leveraged as knowledge assets (KA). However, there is a serious problem "herding" KA into a form that results in leveragable knowledge. Bringing content into a focus that is meaningful to a diverse but specialized audience of users, even within a limited company domain is tough because the language of the content is so messy.

So, what does this have to do with taxonomies and enterprise search, and how they factor into leveraging KA? Taxonomies have a role as a device to promote and secure the meaningful retrievability of content when we need it most or fastest, just-in-time retrieval. If no taxonomies exist to pre-collocate and contextualize content for an audience, we will be perpetually stuck in a mode of having to do individual human filtering of excessive search results that come from "keyword" queries. If we don't begin with taxonomies for helping search engines categorize content, we will certainly never get to the holy grail of semantic search. We need every device we can create and sustain to make information more findable and understandable; we just don't have time to both filter and read, comprehensively, everything a keyword search throws our way to gain the knowledge we need to do our jobs.

Experts recognize that organizing content with pre-defined terminology (aka controlled vocabularies) that can be easily displayed in an expandable taxonomic structure is a useful aid for a certain type of searcher. The audience for navigated search is one that appreciates the clustering of search results into groups that are easily understood. They find value in being able to move easily from broad concepts to narrower ones. They especially like it when the categories and terminology are a close match to the way they view a domain of content in which they are subject experts. It shows respect for their subject area and gives them a level of trust that those maintaining the repository know what they need.

Taxonomies, when properly employed, serve triple duty. Exposing them to search engines that are capable of categorizing content puts them into play as training data. Setting them up within content management systems provides a control mechanism and validation table for human assigned metadata. Finally, when used in a navigated search environment, they provide a visual map of the content landscape.

U.S. businesses are woefully behind in "getting it;" they need to invest in search and surrounding infrastructure that supports search. Comments from a recent meeting I attended reflected the belief that the rest of the world is far ahead in this respect. As if to highlight this fact, a colleague just forwarded this news item yesterday. "On February 13, 2008, the XBRL-based financial listed company taxonomy formulated by the Shanghai Stock Exchange (SSE) was "Acknowledged" by the XBRL International. The acknowledgment information has been released on the official website of the XBRL International (http://www.xbrl.org/FRTaxonomies/)....".

So, let's get on with selling the basic business case for taxonomies in the enterprise to insure that the best of our knowledge assets will be truly findable when we need them.

Search Engines Under the Hood

This week's thoughts come from the pile of serendipitous reading that routinely piles up on my desk. In this case a short article in Information Week caught my eye because it featured the husband of a former neighbor, Ken Krugler, co-founder of Krugle. I'd set it aside because a fellow, David Eddy, in my knowledge management forum group keeps telling us that we need tools to facilitate searching for old but still useful source code. In order to do it, he believes, we need an investment in semantic search tools that normalize the voluminous language variants scattered throughout source code. That would enable programmers to find code that could be re-purposed in new applications.

Now, I have taken the position that source code is just one set of intellectual property (IP) asset that is wasted, abandoned and warehoused for technology archaeologists of centuries hence. I just don't see a solid business case being made to develop search tools that will become a semantic search engine for proprietary treasure troves of code.

Enters old acquaintance Ken Krugler with what seems to be, at first glance, a Web search system that might be helpful for finding useful code out on the Web, including open source. I have finally visited his Web site and I see language and new offerings that intrigue me. "Krugle Enterprise is a valuable tool for anyone involved in software development. Krugle makes software development assets easily accessible and increases the value of a company's code base. By providing a normalized view into these assets, wherever they may be stored, Krugle delivers value to stakeholders throughout the enterprise." They could be onto something big. This is a kind of enterprise search I haven't really had time to think about but may-be I will now.

One thing leading to another, I checked out Ken Krugler's blog and saw an earlier posting: Is Writing Your Own Search Engine Hard? This is recommended reading for anyone who even dabbles in enterprise search technology but doesn't want to get her/his hands dirty with the mechanics. It is short, to-the-point and summarizes how and why so many variations of search are battling it out in the marketplace.

I don't want end-users to struggle too much with the under the hood details but when you are thinking about enterprise search for your organization, it is worth considering how much technology you are getting for the value you want it to deliver, year after year, as your mountains of IP content accrue. Don't give this idea short shrift because search is an investment that keeps giving if it is chosen appropriately for the problem you need to solve.

Search Behind the Firewall aka Enterprise Search

Called to account for the nomenclature “enterprise search,” which is my area of practice for The Gilbane Group, I will confess that the term has become as tiresome as any other category to which the marketplace gives full attention. But what is in a name, anyway? It is just a label and should not be expected to fully express every attribute it embodies. A year ago I defined it to mean any search done within the enterprise with a primary focus of internal content. “Enterprise” can be an entire organization, division, or group with a corpus of content it wants to have searched comprehensively with a single search engine.

A search engine does not need to be exclusive of all other search engines, nor must it be deployed to crawl and index every single repository in its path to be referred to as enterprise search. There are good and justifiable reasons to leave select repositories un-indexed that go beyond even security concerns, implied by the label “search behind the firewall.” I happen to believe that you can deploy enterprise search for enterprises that are quite open with their content and do not keep it behind a firewall (e.g. government agencies, or not-for-profits). You may also have enterprise search deployed with a set of content for the public you serve and for the internal audience. If the content being searched is substantively authored by the members of the organization or procured for their internal use, enterprise search engines are the appropriate class of products to consider. As you will learn from my forthcoming study, Enterprise Search Markets and Applications: Capitalizing on Emerging Demand, and that of Steve Arnold (Beyond Search) there are more than a lot of flavors out there, so you’ll need to move down the food chain of options to get it right for the application or problem you are trying to solve.

OK! Are you yet convinced that Microsoft is pitting itself squarely against Google? The Yahoo announcement of an offer to purchase for something north of $44 billion makes the previous acquisition of FAST for $1.2 billion pale. But I want to know how this squares with IBM, which has a partnership with Yahoo in the Yahoo edition of IBM’s OmniFind. This keeps the attorneys busy. Or may-be Microsoft will buy IBM, too.

Finally, this dog fight exposed in the Washington Post caught my eye, or did one of the dogs walk away with his tail between his legs? Google slams Autonomy – now, why would they do that?

I had other plans for this week’s blog but all the Patriots Super Bowl talk puts me in the mode for looking at other competitions. It is kind of fun.

Search Adoption is a Tricky Business: Knowledge Needed

Enterprise search applications abound in the technology marketplace, from embedded search to specialized e-discovery solutions to search engines for crawling and indexing the entire intranet of an organization. So, why is there so much dissatisfaction with results and heaps of stories of buyer’s remorse? Are we on the cusp of a new wave of semantic search options or better ways to federate our universe of content within and outside the enterprise? Who are the experts on enterprise search anyway?

You might read this blog because you know me from the knowledge management (KM) arena, or from my past life as the founder of an integrated enterprise library automation company. In the KM world a recurring theme is the need to leverage expertise, best done in an environment where it is easy to connect with the experts but that seems to be a dim option in many enterprises. In the corporate library world the intent is to aggregate and filter a substantive domain of content, expertise and knowledge assets on behalf of the specialized interests of the enterprise, too often a legacy model of enterprise infrastructure. Librarians have long been innovators at adopting and leveraging advanced technologies but they have also been a concentrating force for facilitating shared expertise. In fact, special librarians excel at providing access to experts.

We are drowning in technological options, not the least of which is enterprise search and its complexity of feature laden choices. However, it is darned hard to find instances of full search tool adoption or users who love the search tools they are delivered on their intranets. So, I am adopting my KM and library science modes to elevate the discussion about search to a decidedly non-technical conversation.

I really want to learn what you know about enterprise search, what you have learned, discovered and experienced over the past two or three years. This blog and the work I do with The Gilbane Group is about getting readers to the best and most appropriate search solutions that can make positive contributions in their enterprises. Knowing who is using what and where it has succeeded or what problems and issues were encountered is information I can use to communicate, in aggregate, those experiences. I am reaching out to you and those you refer to complete a five minute survey to open the door to more discussion. Please use this link to participate right now Click Here to take survey. You will then have the option to get the resulting details in my upcoming research study on enterprise search.

Just to prove that I still follow exciting technologies, as well, I want to relay a couple of new items. First is a recent category in search, “active intelligence,” adopted as Attivio’s tag line. This is a start-up led by Ali Riaz and officially launched this week from Newton, MA. Then, to get a steady feed of all things enterprise search from guru Steve Arnold, check out his new blog, a lead up to the forthcoming Beyond Search: What to Do When Your Search Engine Doesn’t Work to be published by The Gilbane Group. You’ll be transported from the historical, to the here and now, to the newest tools on his radar screen as you page from one blog entry to another.

Nothing Like a Move by Microsoft to Stir up Analysis and Expectations

Since I weighed in last week on the Microsoft acquisition of FAST Search & Transfer, I have probably read 50+ blog entries and articles on the event. I have also talked to other analysts, received emails from numerous search vendors summarizing their thoughts and expectations about the enterprise search market and had a fair number of phone calls asking questions about what it means. The questions range from “Did Microsoft pay too much?, to “Please define enterprise search,” to “What are the next acquisitions in this market going to be going to be?” My short and flippant answers would be “No,” “Do you have a few hours?” and “Everyone and no one.”

I have seen some excellent analysis contributing relevant commentary to this discussion, some misinterpretation of what the distinction’s are between enterprise search and Web search, and some conclusions that I would seriously debate. You’ll forgive me if I don’t include links to the pieces that influenced the following comments. But one by Curt Monash in his piece on January 14 summarized the state of this industry and its already long history. It is noteworthy that while the popular technology press has only recently begun to write about enterprise search, it has been around for decades in different forms and in a short piece he manages to capture the highlights and current state.

Other commentary seems to imply that Microsoft is not really positioning itself to compete with Google because Google is really about Web (Internet) searching and Microsoft is not. This implies that FAST has no understanding of Web searching. Several points must be made:
1. FAST Search & Transfer has been involved in many aspects of search technologies for a decade. Soon after landing on our shores it was the search engine of choice for the U.S. government’s unifying search engine to support Internet-based searching of agency Web sites by the public. Since then it has helped countless enterprises (e.g. governments, manufacturers, e-commerce companies) expose their content, products and services via the Web. FAST knows a lot about how to make Web search better for all kinds of applications and they will bring that expertise to Microsoft.
2. Google is exploiting the Web to deliver free business software tools that directly challenge Microsoft stronghold ( e.g. email, word processing). This will not go unanswered by the largest supplier of office automation software.
3. Google has several thousand Google Enterprise Search Appliances installed in all types of enterprises around the world, so it is already as widely deployed in enterprises in terms of numbers as FAST, albeit at much lower prices and for simpler application. That doesn’t mean that they are not satisfying a very practical need for a lot of organizations where it is “good enough.”
For more on the competition between the two check this article out.

Enterprise search has been implied to mean only search across all content for an entire enterprise. This raises another fundamental problem of perception. Basically, there are few to no instances of a single enterprise search engine being the only search solution for any major enterprise. Even when an organization “standardizes” on one product for its enterprise search, there will be dozens of other instances of search deployed for groups, divisions, and embedded within applications. Just two examples are the use of Vivisimo now used for USA.gov to give the public access to government agency public content, even as each agency uses different search engines for internal use. Also, there is IBM, which offers the OmniFind suite of enterprise search products, but uses Endeca internally for its Global Services Business enterprise.

Finally, on the issue of expectations, most of the vendors I have heard from are excited that the Microsoft announcement confirms the existence of an enterprise search market. They know that revenues for enterprise search, compared to Web search, have been miniscule. But now that Microsoft is investing heavily in it, they hope that top management across all industries will see it as a software solution to procure. Many analysts are expecting other major acquisitions, perhaps soon. Frequently mentioned buyers are Oracle and IBM but both have already made major acquisitions of search and content products, and both already offer enterprise search solutions. It is going to be quite some time before Microsoft sorts out all the pieces of FAST IP and decides how to package them. Other market acquisitions will surely come. The question is whether the next to be acquired will be large search companies with complex and expensive offerings bought by major software corporations. Or will search products targeting specific enterprise search markets be a better buy to make an immediate impact for companies seeking broader presence in enterprise search as a complementary offering to other tools. There are a lot of enterprise search problems to be solved and a lot of players to divvy up the evolving business for a while to come.

A Call for Papers and Microsoft creates a FAST Opening in the New Year

I closed 2007 with some final takeaways from the Gilbane Conference and notes about semantic search. Already we are planning for Gilbane San Francisco and you are invited to participate. There is no question that enterprise search, in all its dimensions, will be a central theme of several sessions at the conference, June 17th through 19th. I will lead with a discussion in which a whole range of search topics, technologies and industry themes will be explored in a session featuring guest Steve Arnold, author of Google Version 2.0, The Calculating Predator. To complement the sessions, numerous search technology vendors will be present in the exhibit hall.

A most important conference component will be a highlight for conference goers, shared-experiences about selecting, implementing and engaging with search tools in the enterprise. Everyone wants to know what everyone else is doing, learning and what they know about enterprise search. You may want to present your experiences or those of your organization. If you are interested, considering presenting, know of a good case study, usability or “lessons learned” from implementing search technology, please raise your hand. You can do this by reaching out through this link to submit a proposal and make reference to the “enterprise search blog call for papers.” You can be sure I’ll follow-up soon to explore the options for you or a colleague to participate. This is a great opportunity to be part of a community of practitioners like you and attend a conference that always has substantive value for participants.

Leave it to Microsoft to end the year with a big announcement and open the next one with an even bigger one. We knew that the world of enterprise search was going to contract in terms of the number of established vendors, even though it is expanding in new and innovative offerings. Microsoft had to make a bold play in an industry where Google has been the biggest player on the WWW stage while reaching deeper into the enterprise, tickling at Microsoft’s decades-old hold on content creation and capture. So, with the acquisition of FAST Search & Transfer, whose technology may not be the best in the enterprise search market but is certainly the most widely deployed at the high-end, Microsoft opens with a direct challenge to its largest competitor.

Boy! Have the emails been flying this morning. At least I know there will be plenty of material to ponder in the next few weeks and months. P.S. Don’t miss the action in San Francisco!

Enterprise Search and Its Semantic Evolution

That the Gilbane Group launched its Enterprise Search Practice this year was timely. In 2007 enterprise search become a distinct market force, capped off with Microsoft announcing in November that it has definitively joined the market.

Since Jan. 1, 2007, I have tried to bring attention to those issues that inform buyers and users about search technology. My intent has been to make it easier for those selecting a search tool while helping them to get a highly satisfactory result with minimal surprises. Playing coach and lead champion while clarifying options within enterprise search is a role I embrace. It is fitting then, that I wrap up this year with more insights gained from Gilbane Boston; these were not previously highlighted and relate to semantic search.

The semantic Web is a concept introduced almost ten years ago reflecting a vision of how the Worldwide Web (WWW) would evolve. In the beginning we needed a specific address (URL) to get to individual Web sites. Some of these had their own search engines while others were just pages of content we scrolled through or jumped through from link to link. Internet search engines like Alta Vista and Northern Light searched limited parts of the WWW. Then, Yahoo and Google came to provide much broader coverage of all “free” content. While popular search engines provided various categorizing, taxonomy navigation, keyword and advanced searching options, you had to know the terminology that content pages contained to find what you meant to retrieve. If your terms were not explicitly in the content, pages with synonymous or related meaning were not found. The semantic Web vision was to “understand” your inquiry intent and return meaningful results through its semantic algorithms.

The most recent Gilbane Boston conference featured presentations of commercial applications of various semantic search technologies that are contributing to enterprise search solutions. A few high level points gleaned from speakers on analytic and semantic technologies follow.

> Jordan Frank on blogs and wikis in enterprises articulated how they add context by tying content to people and other information like time. Human commentary is a significant content “contextualizer,” my term, not his.
> Steve Cohen and Matt Kodama co-presented an application using technology (interpretive algorithms integrated with search) to elicit meaning from erratic and linguistically difficult (e.g. Arabic, Chinese) text in the global soup of content.
> Gary Carlson gave us understanding of how subject matter expertise contributes substantively to building terminology frameworks (aka “taxonomies”) that are particularly meaningful within a unique knowledge community.
> Mike Moran helped us see how semantically improved search results can really improve the bottom line in the business sense in both his presentation and later in his blog, a follow-up to a question I posed during the session.
> Colin Britton described the value of semantic search to harvest and correlate data from highly disparate data sources needed to do criminal background checks.
> Kate Noerr explained the use of federating technologies to integrate search results in numerous scenarios, all significant and distinct ways to create semantic order (i.e. meaning) out of search results chaos.
> Bruce Molloy energized the late sessions with his description of how non-techies can create intelligent agents to find and feed colleagues relevant information by searching in the background in ways that go far beyond the typical keyword search.
> Finally, Sean Martin and John Stone co-presented an approach to computational data gathering and integrating the results in an analyzed and insightful format that reveals knowledge about the data, not previously understood.

Points taken are that each example represents a building block of the semantic retrieval framework we will encounter on the Web and within the enterprise. The semantic Web will not magically appear as a finished interface or product but it will become richer in how and what it helps us find. Similar evolutions will happen in the enterprise with a different focus, providing smarter paths for operating within business units.

There is much more to pass along in 2008 and I plan to continue with new topics relating to contextual analysis, the value, use and building of taxonomies, and the variety of applications of enterprise search tools. As for 2007, it’s a wrap.

Enterprise Search: Leveraging and Learning From Web Search and Content Tools

Following on my last post in which I covered the unique value propositions offered by a variety of enterprise search products, this one takes a look at the evolution of enterprise search. The commentary by search company experts, executives, and analysts indicates some evolutionary technologies and the escalation of certain themes in enterprise search. Furthermore, the pursuit of organizations to strengthen the link between searching technologies and knowledge enablers has never been more prominently featured taking search to a whole new level beyond mere retrieval.

The following paraphrased comments from the Enterprise Search Keynote session are timely and revealing. When I asked, Will Web and Internet Search Technologies Drive the Enterprise (Internal) Search Tool Offerings or Will the Markets Diverge?, these were some thoughts from the panelists.

Matt Brown, Principal Analyst from Forrester Research, commented that enterprise search demands much different and richer content interpretation types of search technologies. What Web-based searching does is create such high visibility for search that enterprises are being primed to adopt it, but only when it comes with enhanced capabilities.

Echoing Matt’s remarks, Oracle search solution manager Bob Bocchino commented on the difficulty of making search operate well within the enterprise because it needs to deal with structured database content and unstructured files, while also applying sophisticated security features that let only authorized viewers see restricted content. Furthermore, security must be deployed in a way that does not degrade performance while supporting continuous updates to content and permissions.

Hadley Reynolds, VP & Director of the Center for Search Innovation at Fast Search & Transfer, noted that the Web isn’t really making a direct impact on enterprise search innovation but many of the social tools found on the Web are being adopted in enterprises to create new kinds of content (e.g. social networks, blogs and wikis) with which enterprise search engines must cope in richer contextual ways.

Don Dodge, Director of Business Development for the Emerging Business Team at Microsoft further noted that the Internet’s biggest problem is scale. That is a much easier problem to solve than in the enterprise where user standards for what qualifies as a good and valuable search results are much higher, therefore making the technology to deliver those results more difficult.

Among the other noteworthy comments in this session was a negative about taxonomies. The gist of it was that they require so much discipline that they might work for a while but can’t really be sustained. If this attitude becomes the norm, many of the semantic search engines which depend on some type of classification and categorization according to industry terminologies or locally maintained lists will be challenged to deliver enhanced search results. This is a subject to be taken up in a later blog entry.

A final conclusion about enterprise search was a remark about the evolution of adoption in the marketplace. Simply put, the marketplace is not monolithic in its requirements. The diversity of demands on search technologies has been a disincentive for vendors to focus on distinct niches and place more effort on areas like e-commerce. This seems to be shifting, especially with all the large software companies now seriously announcing products in the enterprise search market.

Case Studies: Enterprise Search Success Stories

It has been a week since the annual Gilbane Boston 2007 Conference closed and I am still searching for the most important message that came out of Enterprise Search and Semantic Web Technology sessions. There were so many interesting case studies that I’ll begin with a search function that illustrates one major enterprise search requirement – aggregation.

Besides illustrating a business case for aggregating disparate content using search, the case studies shared three themes:
> Search is just a starting point for many business processes
> While few very large organizations present all of their organization’s content through a single portal, the technology options to manage such an ideal design are growing and up to supporting entire enterprises
> All systems were implemented and operational for delivering value in less than one year, underscoring the trend toward practical and more out-of-the box solutions

Here is a brief take on what came out of just the first two of seven sessions.
Small-medium solutions:
> Use of ISYS to manipulate search results and function as a back-office data analysis tool for DirectEDGAR, the complete SEC filings, presented by Prof. Burch Kealey of the University of Nebraska. Presentation
> Support for search by serendipity across the shareable content domains of members of a trade association (ARF) by finding results that satisfy the searcher in his pursuit of understanding with Exalead, presented by Alain Heurtebise CEO of Exalead. Presentation
> A knowledge portal enabling rapid and efficient retrieval of the complete technical documentation for field service engineers at Otis Elevator to meet rapid response goals when supporting customers using a customized implementation of dtSearch, presented by project consultant Rob Wiesenberg of Contegra Systems, Inc. Presentation

Large solutions calling for search across multi-million record domains:
> Hosted Vivisimo solution federating over 40 million documents across 22,000 government web sites accessible with search results clustered; it records over a half million page views per day on http://USA.gov and was deployed in 8 weeks, presented by Vivisimo co-founder Jerome Pesenti. Presenation
> Intranet knowledge portal for improving customer services by enabling access to internal knowledge assets (over half a million customer cases with all their associated documents) at USi (an AT&T company) using Endeca, a search product USi had experience deploying and hosting for very large e-commerce catalogs, presented by development leader Toby Ford of USi. With one developer it was running in six months. Presentation
> Within a large law firm (Morrison Foerster) and the legal departments of two multi-national pharmaceutical companies (Pfizer and Novartis), Recommind aggregates and indexes content for numerous internal application repositories, file shares and external content sources for unified search across millions of documents, contributing a direct ROI in saved labor by ensuring that required documents are retrieved in a single search process. Presentation

In each of these cases, content from numerous sources was aggregated through the crawling and indexing algorithms of a particular search engine pointed at a bounded and defined corpus of content, with or without associated metadata to solve a particular business problem. In each case, there were surrounding technologies, human architected design elements, and interfaces to present the search interface and results for a predefined audience. This is what we can expect from search in the coming months and years, deployments to meet specialized enterprise needs, an evolving array of features and tools to leverage search results, and a rapid scaling of capabilities to match the explosion of enterprise content that we all need to find and manipulate to do our jobs.

Next week, I will reconstruct more themes and messages from the conference.

Leading Enterprise Initiatives or Reacting to Crisis

My theme leading into the Gilbane Boston Conference this week comes straight from the headlines and New Hampshire political ads that manage to spill over the border into our fair Commonwealth of Massachusetts. If you live outside of the zone of early caucus and primary states, you are probably spared the ad nauseam recitations of all the crises that Rudy Giuliani has met and conquered. In thinking about our collective longing for a true leader in the White House, I began to reflect on all the other places I would like to see leadership. My musings brought me straight to a message I try to impart to clients and professional colleagues struggling with issues of leveraging knowledge and technology.

True leadership is very hard because it requires thinking, projecting and anticipating. It requires abstract thinking about possibilities for making improvements in complex areas. It requires the ability to mentally juggle huge numbers of variables, many of which the true leader knows he/she can’t possibly control but may be able to foresee as possible complications. It requires bucking the status quo.

Anyone can react, and many can react with reasonably appropriate actions, actions that work for the immediate crisis. However, sizing up an enterprise in which things are running in a seemingly routine fashion, and taking the initiative to systematically seek out lurking crises, potential problems, and areas for improvement, and then applying thoughtful and incremental change activities to ensure better outcomes may seem boring – but this is true leadership.

Finally, think about all the ways in which our political leaders seem to thrive on talking about only the monumental crises of the country and world. Think about how our news is driven by immediate crises. We seem to be conditioned to only react to what we are being shown and told. True leaders are seekers, self-educators, investigators, learners and thinkers. Our best leaders are those who get to the core of our political and business enterprises and find a better way for the whole to work more smoothly, with an ultimate goal of bringing positive good to the members of the community. They succeed though personal diligence, finding the will to persevere while immersing themselves in the mundane and routine operations of their domains. They observe and they think about what they observe; they also talk to others and reflect mindfully on what they hear before acting.

As I prepare my opening remarks for several sessions on enterprise search and semantic technologies at the conference over the next three days, I am pondering how I can stimulate the audience to take the time to open their minds to think about what speakers and exhibitors introduce. I want them to think, really think, about what they are hearing. I want them to develop new ideas, new ways of innovating, new ways to make the mundane better and take it back to their enterprises with a purpose – not just with information to be used in the event of a direct work challenge, demand or crisis. I want to lead others to lead from a thoughtfully critical point of view. So, take a look at technologies from the perspective of action toward systemic improvements instead of a reaction to solving only the latest crisis in your enterprise.

Surrounding and Supporting Enterprise Search

In a week when the KMWorld and Enterprise Search Summit were running concurrently in San Jose, Microsoft made an enterprise search product announcement that was actually a well-kept secret for Microsoft. There was plenty of other new product news floating about the marketplace, too. Mark Logic, MuseGlobal, Cognos, SchemaLogic, and Brainware all had their own announcements.

Between November 6 and November 13, these five companies had interesting news to share. The announcements all related to leveraging enterprise content in tandem with search engines. This underscores a strong trend in software product deployment, specifically, that much of it is being rolled out in partnerships in highly heterogeneous environments. While Microsoft’s announcement about free Search Server 2008 Express establishes them as the last major software company to adopt search as a platform, the other technology announcements remind us that integration activity is a core operational consideration and even a necessity for gaining value from search.

In order to tie all the bits and pieces of content across the enterprise into a tidy bundle for simple retrieval, or in order for content to really bring value to solving business problems, it needs packaging. It needs to be packaged at the front end so that search engines can grab useful context and metadata for smarter indexing. It also needs to be well packaged at the output end to present results meaningfully for a particular audience or purpose.

Here is a quick look at what these five complementary technologies do for search plus a link to each of their latest announcements:

> Brainware – combines data capture with a content extraction and distillation learning engine for enhancing categorization relevancy in preparation for natural language queries. It will be embedded in search for a leading enterprise library system, Sirsi/Dynix.
> Cognos – a leading Business Intelligence (BI) software company is being acquired by IBM, whose search products are often paired with Cognos.
> Mark Logic – is a company with an XML content server platform for managing or converting content in XML formats. They just announced MarkMail, a community-focused searchable message archive service, which stores emails as XML documents. Expect more from them on this front.
> MuseGlobal – offers solutions that integrate content from multiple search engines. They just announced availability for presenting results in a fully unified and consistent format from multiple search engines in a SharePoint portal interface.
> SchemaLogic – specializes in content and document type modeling, metadata and vocabulary management using SchemaServer. In the past two weeks they have announced integration with SharePoint to manage metadata. A webinar this week described the interplay with Documentum for document production and retrieval using the FAST search engine.

And what do the other enterprise search vendors have to say about the “surprise” Microsoft announcement? Comments ranged from “we knew it was just a matter of time before they announced” to “good for business, enterprise search is officially now a market.” To the first comment I say, “Not so fast.” For several years rumors have been floated about the imminent acquisition of any number of search companies by MS but nothing materialized. Yes, Microsoft was doing something about enterprise search but until last week “what” was still the question. To the latter I say, “We’ve had an enterprise search market for several years, Microsoft just wanted to be sure it was well established before joining the club.” That was smart of them; let others lay the foundation for a growth industry. It also looks like this is a leveling of the field with Google already playing in Microsoft’s backyard in the free office tools area.

Now the positioning really begins.

Gearing up for Gilbane Boston 2007

I am in a mode of indecision about prioritizing a lot of news in the enterprise search space; it all seems important because we have an agenda focused on search at the upcoming conference on November 27 – 29 in Boston. The following, in no particular order, is not an exhaustive list but representative of happenings in the past month that will surely be the subject of much commentary and discussion by our speakers and panelists:

> Don Dodge of Microsoft, one of our panelists in the search keynote session, is taking on Google’s customer support positioning as an enterprise solutions provider in his blog.
> MondoSoft went from being shut down by its investors to being acquired by possibly two companies (one acquiring MondoSearch and the other acquiring Ontolica) and now both are being acquired by one company. Meanwhile, IntelliSearch is making an offer to MondoSearch clients to “switch and save.”
> Steve Arnold is continuing his drumbeat on Google search patents and their significance.
> Oracle is getting very serious about search as you will hear in this Webinar download and is positioning itself for a holistic approach to managing content in the enterprise.
> Fast and Autonomy are making acquisitions, too, and have begun to act like they are not the only search options for the enterprise by re-focusing their marketing.
> Companies like Connotate, ISYS, Exalead, Recommind, SchemaLogic and Coveo are acquiring good clients and showing their strengths in important niche markets with new enhancements
> Vivisimo’s social search is getting a lot of positive press, inviting a lot of blogging and has set the bar high for competitors to match their offerings.
> Endeca continues to expand its staff, client-base and well-engineered product line; they are also building important alliances with technology and business partners when it makes good sense to do so
> IBM is talking up the potential of semantic search in the enterprise.

Whew!

Most of the above mentioned will be making an appearance or two at GilbaneBoston, as speakers or exhibitors, or both. I am trying to figure out how to make sure the seven sessions on search and semantic Web technologies touch most of the bases but with so much afoot in the search arena, we will be working overtime. If you are going to be an attendee in any capacity, I hope you’ll blog or make comments when I do. We want to hear what you think and learn about from the experts and users alike. There are sure to be surprises. Your take on the programs will be of interest to many. If you do make the conference, be sure to find me and introduce yourself so we can have a chat.

Eating Your Own Dog Food

I sat in on a demonstration for an enterprise search engine this week. There were several things that really impressed me. First it was not canned; although the demonstrators were clearly following a script, and members of the audience members were asking questions throughout the demonstration. The audience was a mix of consultants and prospective customers; they had a lot of questions, asked to be shown features, functions and “what ifs.” The person demonstrating was very soon off script and doing nothing more than answering questions and demonstrating completely ad hoc searches and with excellent results.

The second impressive aspect was that the entire corpus of content was company information for the vendor. It was not everything they have on their intranet for this public view but enough to see that this is a company that actually uses its own technology throughout the enterprise for all of its divisions. Content from Lotus Notes, SharePoint, Documentum, an employee gallery, emails and files were all there and presented in a clear format for the audience to see and understand.

Finally, what impressed me was the extent to which their content reflected how their search engine was being used as a platform for doing business and working collaboratively, internally. This caused me to reflect back to a presentation for an analyst group earlier in this year by another search company. That company was rolling out a number of features in the business intelligence (BI) space. So, I asked them a couple of questions, “How are you using these tools to manage your own enterprise, doing business intelligence? How are you exploiting internal content to understand the dynamics of a rapidly growing company?" There was a long silence but the answer was along the lines of, we aren’t but we should probably consider it and it takes time to think through a strategy for deploying a good BI solution.

In my experience, if everyone in a company with an enterprise search technology product does not find a use for and evolve with its own offerings, then there is a missed opportunity and a flawed business strategy. This missed opportunity is the terrific feedback the developers will get from that internal use and the evangelism that can come from enthusiastic employees. The flawed business strategy is the huge disconnect that develops between customers and suppliers when supplier employees just never have a customer experience themselves. Search and BI are two functions that every company can benefit from using; why on earth wouldn’t every search vendor be aggressively seeking opportunities to apply its own tools.

So, buyers be aware – look for evidence that the purveyors of search actually have their own experiences to talk about and demonstrate. That can tell you a whole lot about how you will be treated as a customer because they are one, too.

Collaboration and Expertise Bring Focus to Enterprise Search

The topic of the month seems to be “social search;” I confess to being a willing participant in this new semantic framing of a rash of innovative new tools for enterprise search products. I would, however, defer to the professional intent of some great new features by stressing that this is really a next step in bringing collaboration closer to where expert knowledge workers do their work. As I view enterprises with a heavy research component, 10 – 30% of the average professional’s time is spent in a search environment. In other words, we all spend a lot of our day just looking for “stuff.” We also spend a significant amount of time in meetings, exchanging emails, and making presentations. More and more of us contribute to collaboration spaces where we work together on various types of document production.

Putting together the work habits and needs of a time-poor and information-rich community of knowledge workers in a post-processing environment where they can “mash up,” tag and commentate their search discoveries is a natural evolution of search technology. It is remarkable to see how search companies that are serious about the enterprise market (search within and for the enterprise) are rapidly turning out enhancements for their audiences, now that they are convinced that “Enterprise 2.0” has a boatload of early adopters in the wings. Search should always be about connecting experts and their content. Add collaboration and the ability to enrich search results by searchers for the benefit of their colleagues and you have a model for, soon-to-be, heavily adopted products.

That pretty much sums up how we should be thinking about “social search” in the enterprise. You can hear more of my views in a KMWorld Webinar, Using Social Search to Drive Innovation through Collaboration next Tuesday in a presentation sponsored by Vivisimo, one of the leaders in this area.

The week had plenty of virtual ink devoted this topic so you might want to check out these two articles with more commentary. The first was in eWeek, by Clint Boulton, Vivisimo Marries Search, Social Networking. The second shows that Google is on the bandwagon, as well, Google Enterprise Search gets social, a blog entry at C|Net News.com by Rafe Needleman.

Trust is a Pact between a Vendor and Its Customer

There is nothing more disappointing to a consultant than to learn that a project in which you gave significant guidance to a client is experiencing a project meltdown…except maybe having everything get off to a positive start only to falter due to problems with the technologies being implemented. I have been burned several times lately and that surprises me because, as a former software vendor myself, I have pretty deep skepticism when it comes to overblown claims and can usually spot the companies I wouldn’t want my clients to trust. This was not one of them.

It is hard to deal with situations that you didn’t consider likely. A big one is a broken promise, even if it is implicit, not explicit. For a vendor to deliver a solid CMS product with a buggy search interface to toggle between keyword and metadata search is one thing. My client spent months getting it to work so that users could seek by keyword or on explicit metadata fields. They rolled it out and it was “OK,” if not great. But after much discussion with the vendor about the bugs, my client was pressured into adopting an upgrade to “solve the problem.” Unfortunately, the upgrade was an experience from hell, but worse was the fact that the old search controls no longer worked and there was no way to search metadata any longer. Having predicated the procurement on being able to search metadata… well, you get the picture.

What happened to the old motto of “first do no harm?” In my world that means you never release an “upgrade” that subtracts functionality. In the words of my client, “we consider this a major regression.” I consider it a serious breach of trust between them and the supplier but also between me and my client. Why would they ever trust my guidance about the solidness of a vendor again? Guess I have my work cut out for me to find some recourse for my client.

On a much more positive note, I will be offering commentary on the subject of trust and technology solutions when I participate in my first Gilbane Webinar with Oracle’s Brian Dirking, Wednesday, October 10th. The title is The Trust Factor: Secure Enterprise Search for High-Value Content and it will include some key considerations when considering your path to a successful search implementation. I’m still optimistic and enthusiastic that you can implement an excellent search solution for your organization if you really chose your strategy, your technology and your business partners carefully and I’m teaming with Oracle to reinforce that message.

It is free, so click on the title to sign up, even if you are in the beginning stages of your quest for a search product. I hope you will join us for the discussion.

About Analysts, Other Naysayers and Search

You have probably noticed a fair degree of skepticism among technology bloggers about search products and search add-ons to other products. There have been quite a few articles lately that generalize “search” into one monolithic group of technologies. You need to really read between the lines to find the “kind of” search that is meaningful to your “kind of” enterprise. If you go back to one of my first blog entries you’ll see I noted that “the market is, frankly, a real mess.” Sue Feldman has been clear that she, too, believes the field to be a “muddle.” Steve Arnold routinely lets us all know how Google’s patent filings suggest a path toward future disruptions in the search marketplace.

Why then would anyone invest in search today? Do it because there are certainly enough really good and appropriate solutions for most enterprises. These may not be in the price range you would like, and may require more overhead support to implement than you think you should need, but you cannot be paralyzed by what the “next big thing might be” because it might not happen for a really long time or at all. You may find a solution today that solves a lot of immediate searching problems for your enterprise and continues to evolve with the needs of your organization.

That said, you need to keep educating yourself and your peers. This article appeared in the paper version of Information Week Aug. 6, 2007 as The Ultimate Answer Machine, but in the online version as The Ultimate Search Engine. It conflates all kinds of search in a single message that the average non-expert buyer wouldn’t be able to sort out. In its product box you have 13 products including Web search and enterprise search mixed together with no differentiation. I’m tracking over 80 products that solve some kind of enterprise search problem and new ones come to my attention every week. Take a look at the online article and be sure to look at reader comments to see more diverse views than just mine. Think about whether what you read makes good business sense.

I could load you up with two dozen interesting articles from just the past week but recently these have caught my attention for more self-education. Check out: Fight Against Infoglut, in Information Week on April 7, 2007, Search technologies for the Internet by Henzinger, Monika. Science, pp. 468-471, 07/27/2007, and Enterprise Search – More than just Google, Analyst Perspectives Consensus Report (these last two are paid content).

Going forward, you will find some interesting perspectives on Oracle’s position in the enterprise search marketplace if you sit in on their Webinar, Oct. 10th. Finally, I hope you can attend the upcoming Gilbane Boston Conference, Nov. 27th – 29th. The case studies and panel discussions will have something for every type of solution. I’ll keep you posted on selected upcoming happenings as they appear; keep your ear to the ground and eyes focused.

Using Expertise and Enterprise Search to Drive New Knowledge

Drew Robb has written an excellent article that reflects new thinking about the convergence of older and emerging technologies. In Search Converging with Business Intelligence for CRM Daily.com on August 28, 2007 I see similarities to my own way of viewing what is possible with newer search tools. Be sure to read it.

In addition to enterprise search, I actively follow enterprise knowledge management. It has been much debated because of confusion about its links to inappropriate technologies and well-intentioned but costly and failed initiatives in many organizations. But, in spite of rumors about its death, KM will remain a boundless frontier of opportunity. At its best, it leverages collaborative and sharing practices to maximize the value of organizations’ discoveries, developments and learning using innovative and often simple practices that work because they suit a particular culture’s way of operating.

Popular writings about business and technology innovation, plus tools and techniques for collaboration and sharing abound. 2007 is surely the year when search has come to dominate the technology landscape as vendors in BI, text mining and text analytics, data management, and countless semantic and Web 2.0 entrants vie to add refinements, and conversely search integrates features from those technologies.

In August I commented on search offerings that have made a point of highlighting their “Sharepoint” connectivity. Similarly, many products are adding claims for exploiting emails. I have long assumed that email should be part of the search engine crawling and indexing mix for any intranet. Given email structure, it seems to have more useful and usable metadata than a lot of other content. Social network analysis tools have been terrific at revealing fascinating relationships and internal communications within organizations, especially in the discovery area as emails have been a source for exploring questionable business practices in legal proceedings.

More sophisticated analytic and semantic techniques for exploiting concepts in content give hints about how technologies can integrate content by mapping experts to the expertise they contribute, even when it is scattered throughout their work, including emails where so many nuggets may reside. An area for development would correlate nuggets of knowledge in emails to reveal hidden and latent expertise, pointing to other content an individual has produced using search with BI and analytics algorithms.

Maybe I’m overreaching but I suspect that a lot of experts may not be sufficiently motivated, disciplined or expected to aggregate their small but useful contributions into more valuable knowledge. Regardless of the reasons for that failure, much could be revealed with the right blending of search, indexing, analytics and business intelligence technologies. The components already exist but implementation to get desired results are not necessarily easy to deploy. A truly innovative expertise exploitation engine would be a knowledge engine of note, able to synthesize new knowledge in unique and interesting ways. Historically, much has been made about the role of serendipity in the “search for truth” and “quest for knowledge.” With the aid of enhanced search technologies to blend any or many expert nuggets, a lot more serendipity might happen.

Mobile Search, Etc. and Early Adoption

It was inevitable that the search market would rapidly expand into the mobile device market and the number of new and established vendors with options for your cell phone or PDA is a daily feast of reading. Combined with other new search options promising voice-enabled search, semantic search, search federating internal with Web and deep Web content, the possibilities for having search served up to suit anyone seem endless.

Tracking over 70 vendors with enterprise search offerings is plenty for me to focus on at the moment. However, as I read the publicity releases and descriptions inviting me to be briefed, in the back of my mind I know that business travelers need access to all kinds of enterprise content regardless of their locale. I worry that my skepticism about mobile search for enterprise content says more about my age than the technology. I might be thinking about struggling with tiny screens and buttons in less than optimal viewing conditions. But I am open to the possibilities and know it will become pervasive, as will all the other flavors of search.

Caution: before jumping into hot new offerings for enterprise search or any other technology, take a deep breath and think about some consequences of being that early adopter.

Security models for enterprise search in which content is being aggregated or federated from across numerous structured and unstructured content repositories is a very big issue, takes lots of planning, mapping, and time to deploy and test. Add the considerations of a start-up vendor dealing with your precious knowledge assets in a wireless world and you might think twice.

Recent CIO Research, CIO Vendor Report Card results for 52 top IT vendors broke out statistics in several areas. The responding audience ranked as their highest priorities two that would be very hard to judge for a new offering:
> (Vendor) Delivers on Promises
> Ongoing Support after the Sale and Implementation

Another statistic that was reported in the CIO Vendor Report Card was a measure of the likelihood that the respondent would be willing to recommend the vendor. This reminds us of a critical issue that early adopters of new products and technologies can’t easily resolve, finding recommenders or product case studies.

I don’t want to pick on mobile search especially; there is so much hot stuff coming down the pipeline that it is easy to get carried away. But a sign to take it slowly is when your own enterprise is struggling to keep up with new releases of products from known companies, and lags behind in fully exploiting technology you already have. The hottest “must haves” won’t even make it through initial deployment in this environment. Making certain that enterprise search is working well within the organization is a necessary and critical first step. Having a vendor with a track record and happy customers is a close second.

Sharepoint and Search

Sharepoint repositories are a prime content target for most search engines in the enterprise search arena, judging from the number of announcements I’ve previewed from search vendors in the last six month. This list is long and growing (Names link to press releases or product pages for Sharepoint search enabling):

> Autonomy
> Coveo
> dtSearch
> FAST
> ISYS
> Longitude from BA-Insight
> Ontolica from Mondosoft
> OpenText
> Oracle
> Recommind
> Schemalogic
> Vivisimo
> X1
> …and surely more I’ve missed

Almost a year ago I began using a pre-MOSS version of Sharepoint to collect documents for a team activity. Ironically, the project was the selection, acquisition, implementation of a (non-Sharepoint) content management system to manage a corporate intranet, extranet, and hosted public Web site. The version of Sharepoint that was “set up” for me was strictly out of the box. Not being a development, I was still able to muddle my way through setting up the site, established users, posting announcements and categories of content to which I uploaded about fifty or sixty documents.

The most annoying discovery was the lack of a default search option. Later updating to MOSS solved the problem but at the time it was a huge aggravation. Because I could not guarantee a search option would appear soon enough, I had to painstakingly create titles with dates in order to give team members a contextual description as they would browse the site. Some of the documents I wanted to share were published papers and reviews of products. Dates were not too relevant for those, so I “enhanced” the titles with my own notations to help the finders select what they needed.

These silly “homemade” solutions are not uncommon when a tool does not anticipate how we would want to be able to use it. They persist as ways to handle our information storage and retrieval challenges. Since the beginning of time humans have devised ways to store things that they might want to re-use at some point in the future. Organizing for findability is an art as much at it is science. Information science only takes one so far in establishing the organizing criteria and assigning those criteria to content. Search engines that rely strictly on the author’s language will leave a lot of relevant content on the shelf for the same reasons as using Dewey Decimal classification without the complementary card catalog of subject topics. The better search engines exploit every structured piece of data or tagged content associated with a document, and that includes all the surrounding metadata assigned by “categorizers.” Categorizers might be artful human indexers or automated processes. Search engines with highly refined, intelligent categorizers to enable semantically rich finding experiences bring even more sophistication to the search experience.

But back to Sharepoint, which does have an embedded search option now, I’ve heard more than one expert comment on the likelihood that it will not be the “search” of choice for Sharepoint. That is why we have so many search options scrambling to promote their own Sharepoint search. This is probably because the organizing framework around contributing content to Sharepoint is so loosey goosey that an aggregation of many Sharepoint sites across the organization will be just what we’ve experienced with all these other homegrown systems – a dump full of idiosyncratic organizing tricks.

What you want to do, thoughtfully, is assess whether the search engine you need will share only Sharepoint repositories OR both structured and unstructured repositories across a much larger domain of types of content and applications. It will be interesting to evaluate the options that are out there for searching Sharepoint gold mines. Key questions: Is a product targeting only Sharepoint sites or diverse content? How will content across many types of repositories be aggregated and reflected organized results displays? How will the security models of the various repositories interact with the search engine? Answering these three questions first will quickly narrow your list of candidates for enterprise search.

The Marginal Influence of E-commerce Search and Taxonomies on Enterprise Search Technologies

As we gear up for Gilbane Boston 2007, the number of possible topics to include in the tracks related to search seems boundless. The search business is in a transitional state but in spite of disarray is still pivotal in its impact on business and current culture. The sessions will reflect the diversity in the market.

One trend is quite clear; the amount of money and effort being expended for Web search or site search on commercial Web sites is a winner in the “search technology” revenues war with annual revenues measuring well into the $billions. On the other hand, a recent Gartner study described the 2006 revenues for enterprise search as below $400M. This figure comes from reading an excellent article, Enterprise Search: Seek and Maybe You’ll Find, by Ben DuPont in Intelligent Enterprise. Check it out.

The distinctions between search on the Web and search within the enterprise are numerous but here are two. First, Internet Web search revenue is all about marketing. Yes, we use it to discover, learn, find facts, and become more informed. But when companies supplying search technology to expose you to their content on the Internet they do so to facilitate commerce. If it falls into the hands of organizations that have other intent, libraries or government agencies, so be it.

As we all know, when we are at work, seeking to discover, learn or find facts to do our jobs better, we need a different kind of search. Thus, we seek a clear search winner built just for our enterprise with all of its idiosyncrasies. The problem is that what is inside does not look like the rest of the world’s content as it is aggregated for commercial views. Enterprises are unique and operate sometimes chaotically, or, at best, with nuanced views of what information is most important.

The second distinction relates to taxonomies, and the increase in their development and use. I’ve seen a dramatic increase in job postings for “taxonomists” and have managed several projects for enterprises over the years to build these controlled lists of terms for categorizing content. What is noteworthy about recent job opportunities is that most seem to be for customer facing Web sites. Historically, organizations with substantial internal content (e.g. research reports, patents, laboratory findings, business documents) hired professionals to categorize materials for a narrowly defined audience of specialists. The terminology was often highly unique, could number in the hundreds or thousands of terms, even for a relatively small enterprise. This is no longer a common practice.

Slow financial growth in enterprise search markets is no surprise. Like many tools designed and marketed for departments not directly tied to revenue generation, search goes begging for solid vertical markets. Search’s companion technologies are also struggling to find a lucrative toehold for use within the organization. Content management systems integrated with rich and efficient taxonomy building and maintenance functions are hard to find.

I am confident that tools in CMS products for building and maintaining complex taxonomies will not improve until enterprises find a solid business reason to put professional human resources into doing content management, taxonomy development, search, and text analytics on their most important knowledge assets. This is a tough business proposition compared to the revenues being driven on the Internet. What businesses need to keep in mind is that without the ability to leverage their internal knowledge content assets better, smarter and faster, there won’t be innovative products in the pipeline to generate commerce. Losing track of your valuable intellectual resources is not a good long term strategy. Once you begin committing to solid content resource management strategies, enterprise technology products will improve to meet your needs.

Search and Need

Since an attempt to parse, in the simplest terms, the “enterprise search” market in January, I have been exposed to no less than 77 products and vendors whose offerings have been brought to my attention. Add to that another 20 or 30 peripheral offerings in the text mining and text analytics sphere and you’ll understand why the need for a focused view when considering products.

Selling and marketing at its best sells to a need. Need expresses something about users, user behaviors, user requirements, and problems to be solved. Need also implies emotions and that may present a problem when it comes to making business decisions. Nothing plays into emotional business decisions like money, as illustrated by one IT manager’s reaction to this week’s Yahoo News story about Google offering its search appliance for small Web sites for $100 for up to 5,000 pages. Noting that $500/year would support up to 50,000 Web pages, he thought it could be a solution for the company’s intranet. In a tough budget situation it seemed to make sense because the maintenance fee for current search software far exceeds $500.

Let’s be clear, Google is offering site search for a Web site on the World-wide Web, not internal enterprise sites. There is a huge difference in the number of variables to be considered not the least of which are:
1. Who is authoring and maintaining the target content, and what do they expect to have the search engine do with the tags and content?
2. Who are the users, what are they looking for, and how do they expect it to be displayed?
3. What is the software providing in the way of managing and supporting metadata?
4. Where is the software going to run and be maintained?
5. What are the security and authorization considerations?
6. What about all the internal content that is not “Web pages” (e.g. PDFs, spreadsheets, slide shows, images) with their associated metadata that may not be supported in this license but are fundamental to an enterprise search solution?
7. What do page ranking and ad management have to do with internal search requirements?

Just to be clear, there are other solutions that may come with levels of Web site search support that are more suited to many small organizations, internal and external. This week I learned more about one such offering, PicoSearch that has options from free to very reasonable monthly charges bundled with service for hosting search for an organization’s content. It can also provide some levels of password protection and security controls. This may not be an optimal choice for organizations with complex and multi-faceted search interfaces but could be perfect for associations, educational institutions, and small businesses with straightforward product lines.

Keep in mind, inexpensive does not mean “cheap” and it is also not the first qualifying criteria for what is “appropriate.”

Random Notes from the World on Search

A week late I am wrapping up my first six months blogging for The Gilbane Group on enterprise search. I am attempting a retrospective of discoveries, thoughts and issues that surfaced in second quarter. June was especially busy and now that I have had time to sort the sortable here are a few noteworthy highlights and reflections on them. In short, the search market is complex and becoming more so on a monthly basis.

Google the company and Google the product suite are so dominant that any article about search in the mainstream or technical presses evokes the “G-word.” This happens even if Google is not the main topic. Consider for example Walter Mossberg in the Wall Street Journal in this article June 28, “Ask.com Takes Lead in Designing Display of Search Results.” The first paragraph never mentioned Ask.com but began “Google and other search companies …” On the same page was an article “Start-ups Make Inroads with Google’s Work Force.” Earlier that week the New York Times ran a story “The Human Touch that May Loosen Google’s Grip,” MassHighTech referred to Google throughout an article “Why the Best Search Marketers are Right-Brained,” and Intelligent Enterprise did as well in “Enterprise Search: Seek and Maybe You’ll Find.” [More about the latter further on.] A search on www.Clusty.com for “Google” under News>Top News today gets 89 hits and “Toyota” 49.

Korea presents us with a take on Internet search that I think is highly relevant to the enterprise search market as described in the New York Times, July 5 in “South Koreans Connect Through Search Engine.” It turns out that the amount of content in Korean on the WWW is so scanty that Google is irrelevant. Instead a five year-old company, Naver.com is giving Koreans what they really need, answers to questions native Koreans are seeking, built up collaboratively through their cultural “helpfulness.” Naver.com services 77% of all Internet “searches originating in South Korea.” Just as Google can’t deliver to a Korean population what it wants to know, Google can’t really “understand” all of the information needs nuances in culturally diverse enterprises. Naver maintains “questions and answers in proprietary databases not shared with other portals or search engines” as well an enterprise might want to do.

At the Red Herring conference in Boston on June 28, a panel of industry leaders, in a session entitled “The New Frontier in Search” was asked by the moderator whether there will “be any major breakthroughs in semantic search in the next ten years.” The answer from all four including Jeff Cutler of Answers.com and Doug Leeds of Ask.com was an unequivocal , “NO!” I have a list of over 30 companies working on or publicly “sniffing around” the semantic search marketplace. Others are sure to be engaged in stealth work so “not in ten years” is hard to digest but who really knows?

Also at Red Herring, in an interview with EMC’s Mark Lewis, he emphasized a compelling issue for enterprise search, “security,” namely authentication for permission to view search results. In another panel session moderated by Judy Hurwitz on SOA, the security issue was even more dominant as speakers discussed the complexities of integrating heterogeneous applications in a SOA environment while maintaining security integrity. As the number of variables in the architecture rises, so too the technical difficulties of making secure content really secure in search.

The Enterprise Search landscape is pretty crowded with companies that are more focused on helping us find what is in the organization than what is on an enterprise’s Web site. Summarizing the challenges these vendors face is the aforementioned article, “Enterprise Search: Seek and Maybe You’ll Find.” Their market is my beat but grappling with the realities of serving such diverse audiences is a serious necessity.

OK, this blog entry is already too long but you get the idea. The fact that the New York Times has recently had at least one article a week relating to search technologies is really a business marker. While search was introduced to professional searchers 35 years ago, it has been a real sleeper for most of the decades since. Web technology is truly the enabler of so much that makes search work for the masses in so many environments. It’s pretty clear that although search is ubiquitous in the workplace, its commodity status and the normalizing of enterprise search protocols are still a few years off. It is going to be interesting to see who stumbles and who prevails of the current bumper crop of offerings. Or will another disruption take us into more innovative forms of search?

Stay tuned for the next six months – I’m predicting more shakeout in the industry and more adoption of different flavors of search in more organizations. Trying to keep up will be the primary challenge.

Respect for Complexity and Security are Winners

I participated in one search vendor’s user conference this week, and a webinar sponsored by another. Both impressed me because they expressed values that I respect in the content software industry and they provided solid evidence that they have the technology and business delivery infrastructure to back up the rhetoric.

You have probably noted that my blog is slim on comments about specific products and this trend will continue to be the norm. However, in addition to the general feeling of good will from Endeca customers that I experienced at Endeca Discover 2007, I heard clear messages from sessions I attended that reinforced the company’s focus on helping clients solve complex content retrieval problems. Complexity is inherent in enterprises because of diversity among employees, methods of operating, technologies deployed and varied approaches to meeting business demands at every level.

In presentations by Paul Sonderegger and Jason Purcell care was given to explain Endeca’s approach to building their search technology solutions and why. At the core is a fundamental truth about how organizations and people function; you never know how a huge amount of unpredictably interconnected stuff will be approached. Endeca wants its users to be able to use information as levers to discover, through personalized methods, relationships among content pieces that will pry open new possibilities for understanding the content.

Years ago I was actively involved with a database model called an associative structural model. It was developed explicitly to store and manipulate huge amount of database text and embodied features of hierarchical, networked and relational data structures. It worked well for complex, integrated databases because it allowed users to manipulate and mingle data groups in unlimited ways. Unfortunately, it required a user to be able to visualize the possibilities for combining and expressing data from hundreds of fields in numerous tables by using keys. This structural complexity could not easily be taught or learned, and tools for simple visualization were not available in the early 1980s. As I listened to Jason Purcell describe Endeca’s optimized record store, and concept of “intra-query” to provide solutions for the problems posed by double uncertainty I thought, “They get it.” They have acknowledged the challenge of making it simple to update, use and exploit vast knowledge stores; they are working hard to meet the challenge. Good for them! We all want flexibility to work the way we want but if it is not easy we will not adopt.

In a KMWorld webinar, Vivisimo’s Jerome Pesente and customer Arnold Verstraten of NV Organon co-presented with Matt Brown of Forrester Research. The theme was search security models. Besides the reasons for up-front consideration for security when accessing and retrieving from enterprise repositories, three basic control models were described. All three were based on access control lists (ACLs), how and why they are used by Vivisimo.

Having worked with defense agencies, defense contractors and corporations with very serious security requirements on who can access what, I am very familiar with the types of data structures and indexing methods that can be used. I was pleased to hear the speakers address trade-offs that include performance and deployment issues. It served to remind me that organizations do need to be thinking about this early in the selection process; inability to handle the most sensitive content appropriately should eliminate any enterprise search vendor that tries to equivocate on security. Also, as Organon did, there is nothing that demonstrates the quality of the solution like a “bake-off” against a sufficient corpus of content that will demonstrate whether all documents and their metadata that must not be viewed by some audiences in fact are always excluded from search results for all in those restricted audiences. Test it. Really test it!

Turbo Search Engines in Cars; it is not the whole solution.

In my quest to analyze the search tools that are available to the enterprise, I spend a lot of time searching. These searches use conventional on-line search tools, and my own database of citations that link to articles, long forgotten. But true insights about products and markets usually come through the old-fashioned route, the serendipity of routine life. For me search also includes the ordinary things I do everyday:
> Looking up a fact (e.g. phone number, someone’s birthday, woodchuck deterrents), which I may find in an electronic file or hardcopy
> Retrieving a specific document (e.g. an expense form, policy statement, or ISO standard), which may be on-line or in my file cabinet
> Finding evidence (e.g. examining search logs to understand how people are using a search engine, looking for a woodchuck hole near my garden, examining my tires for uneven tread wear), which requires viewing electronic files or my physical environment
> Discovering who the experts are on a topic or what expertise my associates have (e.g. looking up topics to see who has written or spoken, reading resumes or biographies to uncover experience), which is more often done on-line but may be buried in a 20-year old professional directory on the shelf
> Learning about a subject I want or need to understand (e.g. How are search and text analytics being used together in business enterprises? what is the meaning of the tag line “Turbo Search Engine” on an Acura ad?), which were partially answered with online search but also by attending conferences like the Text Analytics Summit 2007 this week

This list illustrates several things. First search is about finding facts, evidence, aggregated information (documents). It is also about discovering, learning and uncovering information that we can then analyze for any number of decisions or potential actions.

Second, search enables us to function more efficiently in all of our worldly activities, execute our jobs, increase our own expertise and generally feed our brains.

Third, search does not require the use of electronic technology, nor sophisticated tools, just our amazing senses: sight, hearing, touch, smell and taste.

Fourth, that what Google now defines as “cloud computing” and MIT geeks began touting as “wearable” technology a few years ago have converged to bring us cars embedded with what Acura defines as “turbo search engines.” On this fourth point, I needed to discover the point. In small print on the full page ad in Newsweek were phrases like “linked to over 7,000,000 destinations” and “knows where traffic is.” In even tinier print was the statement, “real-time traffic monitoring available in select markets…” I thought I understood that they were promoting the pervasiveness of search potential through the car’s extensive technological features. Then I searched the Internet for the phrase “turbo search engine” coupled with “Acura” only to learn that there was more to it. Notably, there is the “…image-tagging campaign that enables the targeted audience to use their fully-integrated mobile devices to be part of the promotion.” You can read the context yourself.

Well, I am still trying to get my head around this fourth point to understand how important it is to helping companies find solid, practical search solutions to problems they face in business enterprises. I don’t believe that a parking lot full of Acura’s is something I will recommend.

Fifth, I experienced some additional thoughts about the place for search technology this week. Technology experts like Sue Feldman of IDC and Fran Halper of Hurwitz & Associates appeared on a panel at the Text Analytics Summit. While making clear the distinctions between search and text analytics, and text analytics and text mining, Sue also made clear that algorithmic techniques employed by the various tools being demonstrated are distinct for each solving different problems in different business situations. She and others acknowledge that finally, having embraced search, enterprises are now adopting significant applications using text analytic techniques to make better sense of all the found content.

Integration was a recurring theme at the conference, even as it was also obvious that no one product embodies the full range of text search, mining and analytics that any one enterprise might need. When tools and technologies are procured in silos, good integration is a tough proposition, and a costly one. Tacking on one product after another and trying to retrofit to provide a seamless continuum from capturing, storing, and organizing content to retrieving and analyzing the text in it, takes forethought and intelligent human design. Even if you can’t procure the whole solution to all your problems at once, and who can, you do need a vision of where you are going to end up so that each deployment is a building block to the whole architecture.

There is a lot to discover at conferences that can’t be learned through search, like what you absorb in a random mix of presentations, discussions and demos that can lead to new insights or just a confirmation of the optimal path to a cohesive plan.

A Story About Structured Content and Search

Today was spent trying to sift through four distinct piles of paper, a backlog of email messages, and managing my calendar. My goal was first to get rid of documents and articles that are too old or irrelevant to “deal with.” The remainder I intended to catalog in my database, file in the appropriate electronic folder, or place in a new pile as a “to-do list.” This final pile plus my email “In Box” would then be systematically assigned to a spot in my calendar for the next six weeks. I did have deadlines to meet, but they depended on other people sending me content, which never came. So I kept sifting and organizing. As you can guess, the day that began with lofty intentions of getting to the bottom of the piles so that I could prioritize my real work is ending, instead, with this blog entry. It is not the one I began for this week four days ago.

First, the most ironic moment of the day came from the last pile in which I turned over an article that must have made an impression in 1997, from Newsweek it was entitled Drowning in Data. I knew I shouldn’t digress, again, but reading it confirmed what I already knew. We all have been “drowning in data” since at least 1997 and for all the same reasons we were back then, Internet publishing, email, voice mail, and faxes (well not so much anymore). It has the same effect as it did ten years ago on “info-stressed” professionals; it makes us all want to go slower so we can think about what is being thrown at us. Yes, that is why I was isolated trying to bring order to the info-glut on my desks. The article mentioned that “the average worker in a large corporation sent and received an astounding 177 messages a day…”

That is the perfect segue to my next observation. In the course of the day, while looking for emails needed to meet deadlines, I emptied over 300 messages from my Junk Mailbox, over 400 from my Deleted Mailbox, and that left me with just 76 in my In Mailbox, which I will begin acting on when I finish this blog entry. (Well, may-be after dinner.) What happened today that caused six different search vendors to send invitations to Webinars or analyst briefings? Oh well, when I finally get around to filling out my calendar for the next six weeks I will probably find out that some, if not all, conflict with appointments I already have. So, may-be I should finish the calendar before responding to the emails.

In the opening of this story I mentioned four distinct piles; I lied. As one document was replaced by another, I discovered that there was no unifying theme for any one pile. So much for categorization, but I did find some important papers that required immediate action, which I took.

Finally, I uncovered an article from http://techweb.cmp.com/iw in 1996. The Information Week archives don’t go back that far but the title was Library on an Intranet. It described a Web-based system for organizing corporate information by Sequent Computer Systems. I know why I saved it; because I had developed and was marketing corporate library systems to run over company networks back in 1980. I did find a reference to the Sequent structured system for organizing and navigating corporate content. You will find it at: http://www.infoloom.com/gcaconfs/WEB/seattle96/lmd.HTM#N136. It is a very interesting read.

What a ride we have had trying to corral this info-glut electronically for over 30 years. From citation searching using command languages in the 1970s, to navigation and structured searching in library systems in the 1980s and 90s, to Web-based navigation coupled with full-text searching in the mid-90s; it never ends. And I am still trying to structure my paper piles into a searchable collection of content.

May-be browsing the piles isn’t such a bad idea after-all. I never would have found those articles using “search” search.

Postscript: This really happened. When I finished this blog entry and went to place the “Drowning…” article on a pile I never got to, there on the top was an article from Information Week, April 9, 2007, entitled “Too Much Information.” I really didn’t need to read the lecturing subtitle: Feeling overwhelmed? You need a comprehensive strategy, not a butterfly net, to deal with data overload. I can assure you, I wasn’t waving butterfly nets all day.

Reflecting on BI Shifts and Google Big Moves in the Search Market

Sometimes it pays to be behind in reading industry news. The big news last week was Google’s new patents and plans to enhance search results using metadata and taxonomy embedded in content. This was followed by the news that Business Objects plans to acquire Inxight, a Xerox PARC spin-off that has produced a product line with terrific data visualization tools, highly valued in the business analytics (BI) marketplace. I had planned to write about the convergence of the enterprise search and BI markets this week until I caught up with industry news from April and early May. This triggered a couple of insights into these more recent announcements.

In April an Information Week article noted that Google has, uncharacteristically, contributed two significant enhancements to MySQL: improved replication procedures across multiple systems and expanded mirroring. Writer Babcock also noted that “Google doesn’t use MySQL in search” but YouTube does. I believe Google will come to be more tied to MySQL as they begin to deploy new search algorithms that take advantage of metadata and taxonomies. These need good text database structures to be managed efficiently and leveraged effectively to produce quality results from search on the scale that Google does it. Up to now Google results presentation has been influenced more by transaction processing than semantic and textual context. Look for more Google enhancements to MySQL to help it effectively manage all that meaningful text. The open source question is will more enhancements be released by Google for all to use? A lot of enterprises would benefit from being able to depend on continual enhancements to MySQL so they could (continue to) use it instead of Oracle or MS-SQL server as the database back-end for text searching.

The other older news (Information Week, May 7th) was that Business Objects was touting “business intelligence for ‘all individuals’” with some new offerings. BO’s acquisition announcement just last week, that they plan to acquire Inxight, only strengthens their position in this market. Inxight has been on the cusp of BI and enterprise search for several years and this portends more convergence of products in these growing markets. Twenty-five years ago when I was selling text software applications, a key differentiator was strong report building tool sets to support “slicing and dicing” database content in any desired format. It sounds like robust, intuitive reporting tools for all enterprise users of content applications is still a dream but much closer to reality for the high-end market.

With all the offerings and consolidation in BI and search, the next moves will surely begin to push some offerings with search/BI to a price point that small-medium businesses (SMBs) can afford. We know that Microsoft sees the opening (Information Week, May 14th) and let’s hope that others do as well.

Will Steve Arnold Scare IT Into Taking Search in the Enterprise Seriously?

Steve Arnold of ArnoldIT struck twice in a big way last week, once as a contributor to the Bear, Stearns & Co. research report on Google and once as a principal speaker at Enterprise Search in New York. I’ve read a copy of the Bear Stearns report, which contains information that should make IT people pay close attention to how they manage searchable enterprise content. I can verify that this blog summary of Steve’s New York speech by Larry Digman sounds like vintage Arnold, to the point and right on it. Steve, not for the first time, is making points that analysts and other search experts routinely observe about the lack of serious infrastructure vested in making content valuable by enhancing its searchability.

First is the Bear Stearns report, summarized for the benefit of government IT folks with admonitions about how to act on the technical guidance it provides in this article by Joab Jackson in GCN. The report’s appearance in the same week as Microsoft’s acquisition of aQuantive is newsworthy in itself. Google really ups the ante with their plans to change the rules for posting content results for Internet searches. If Webmasters actually begin to do more sophisticated content preparation to leverage what Google is calling its Programmable Search Engine (PSE), then results using Google search will continue to be several steps ahead of what Microsoft is currently rolling out. In other words, while Microsoft is making its most expensive acquisition to tweak Internet searching in one area, Google is investing its capital in its own IP development to make search richer in another. Experience looking at large software companies tells me that IP strategically developed to be totally in sync with existing products have a much better chance of quick success in the marketplace than companies that do acquisitions to play catch up. So, even though Microsoft, in an acquiring mode, may find IP to acquire in the semantic search space (and there is a lot out there that hasn’t been commercialized), its ability to absorb and integrate it in time to head off this Google initiative is a real tough proposition. I’m with Bear Stearn’s guidance on this one.

OK, on to Arnold’s comments at Enterprise Search, in which he continues a theme to jolt IT folks. As, already noted, I totally agree that IT in most organizations is loath to call on information search professionals to understand the best ways to exploit search engine adoption for getting good search results. But I am hoping that the economic side of search, Web content management for an organization’s public facing content, may cause a shift. Already, I am experiencing Web content managers who are enlightened about how to make content more findable through good metadata and taxonomy strategies. They have figured out how to make good stuff rise to the top with guidance from outside IT. When sales people complain that their prospects can’t find the company’s products online, it tends to spur marketing folks to adjust their Web content strategies accordingly.

It may take a while, but my observation is that when employees see search working well on their public sites, they begin to push for equal quality search internally. Now that we have Google paying serious attention to metadata for the purpose of giving search results semantic context, maybe the guys in-house will begin to get it, too.

Mapping Search Requirements

Last week I commented on the richness of the search marketplace. However, diversity presents the enterprise buyer with pressure to be more focused on immediate and critical search needs.

The Enterprise Search Summit is being held in New York this week. Two years ago I found it a great place to see the companies offering search products, where I could easily see them all, and still attend every session in two days. This year there are over 40 exhibitors, most offering solutions for highly differentiated enterprise search problems. Few of the offerings will serve the end-to-end needs of a large enterprise but many would be sufficient for medium to small organizations. The two major search engine categories used to be Web content keyword searching, and structured searching. Not only is my attention as an analyst being requested by major vendors offering solutions for different types of search but new products are being announced weekly. Newcomers include those describing their products as data mining engines, search and reporting “platforms,” BI intelligence engines, semantic and ontological search engines. This mix challenges me to determine if a product really solves a type of enterprise search problem before I pay attention.

You, on the other hand, need to do another type of analysis before considering specific options. Classifying search categories, taking a faceted approach will help you narrow down the field. Here is a checklist for categorizing what and how content needs to be found:

> Content types (e.g. HTML pages, PDFs, images)
> Content repositories (e.g. database applications, content management systems, collaboration applications, file locations)
> Types of search interfaces and navigation (e.g. simple search box, metadata, taxonomy)
> Types of search (e.g. keyword, phrase, date, topical navigation)
> Types of results presentation (e.g. aggregated, federated, normalized, citation)
> Platforms (e.g. hosted, intranet, desktop)
> Type of vendor (e.g. search-only, single purpose application with embedded search, software as service – SaS )
> Amount of content by type
> Number and type of users by need (personas)

Then use any tools or resources at hand to harvest an understanding of the mapping results to learn who needs what type of content, in what format and its criticality to business requirements. Prioritizing the facets produces a multi-dimensional view of enterprise search requirements. This will go a long way to narrowing down the vendor list and gives you a tool to keep discussions focused.

There are terrific options in the marketplace and they will only become richer in features and complexity. Your job is to find the most appropriate solution for the business search problem you need to solve today, at a cost that matches your budget. You also want a product that can be implemented rapidly with immediate benefit linking to a real business proposition.

Will Search Technology Ever Become a Commodity?

This week, EMC announced a collaborative research network, with this headline: New EMC Innovation Network to Harness Worldwide Tech Resources, Accelerate Information Infrastructure Innovation. Among the areas that the research network will explore are Semantic Web, search, context, and ontological views.

There is a lot to feed on in this announcement but the most interesting aspect is the juxtaposition with other hardware giants' forays into the world of document and content search software (e.g. IBM, CISCO), and recent efforts by software leaders Oracle and Microsoft to strengthen their offerings in the area of content and search.

One of the phrases in EMC’s announcement that struck me is the reference to “information infrastructure.” This phrase is used ubiquitously by IT folks to aggregate their hardware and network components with the assumption that because these systems store and transport data, they are information infrastructure. We need to recognize that there are two elements missing from this infrastructure view, skilled knowledge workers (e.g.content structure architects, taxonomists, specialist librarians) and software applications for content authoring, capture, organization, and retrieval. Judging from the language of EMC’s press release this might just be tacit recognition that hardware and networks do not make up an information infrastructure. But those of us in search and content management knew that all along; we don’t need a think tank to show us how the pieces fit together nor even how to innovate to create good infrastructure. Top notch professionals have been doing that for decades. Will this new network really reveal anything new?

EMC does not explicitly announce a plan to make search and information infrastructure product commodities but they do express the desire to build “commercial products” for this market. They have already acquired a few of the software components but have yet to demonstrate a tight integration with the rest of the company. Usually innovation comes from humble roots and grows organically through the sponsorship of a large organization, self-funding or other interested contributors. This effort to lead an innovation community to solutions for information infrastructure has the potential to spawn growth of truly innovative tools, methods and even standards for diverse needs and communities. Alternatively, it may simply be a push to bring a free-wheeling industry of multi-faceted components under central control with the result being tools and products that serve the lowest common denominator users.

From a search point of view, I for one am enjoying the richness of the marketplace and how varied the product offerings are for many specialized needs. For the time being, I remain skeptical that any hardware or software giant can sustain the richness of offerings that get to the heart of particular business search needs in a universal way. Commodity search solutions are a long way off for the community of organizations I encounter.

Turning Around a Bad Enterprise Search Experience

Many organizations have experimented with a number of search engines for their enterprise content. When the search engine is deployed within the bounds of a specific content domain (e.g. a QuickPlace site) the user can assume that the content being searched is within that site. However, an organization’s intranet portal with a free-standing search box comes with a different expectation. Most people assume that search will find content anywhere in the implied domain, and for most of us we believe that all content belonging to that domain (e. g. a company) is searchable.

I find it surprising how many public Web sites for media organizations (publishers) don’t appear to have their site search engines pointing to all the sub-sites indicated in site maps. I know from my experience at client sites that the same is often true for enterprise searching. The reasons are numerous and diverse, commentary for another entry. However, one simple notation under or beside the search box can clarify expectations. A simple link to a “list of searchable content” will underscore the caveat or at least tip the searcher that the content is bounded in some way.

When users in an organization come to expect that they will not find, through their intranet, what they are seeking but know to exist somewhere in the enterprise, they become cynical and distrustful. Having a successful intranet portal is all about building trust and confidence that the search tool really works or “does the job.” Once that trust is broken, new attempts to change the attitudes by deploying a new search engine, increasing the license to include more content, or doing better tuning to return more reliable results is not going to change minds without a lot of communication work to explain the change. I know that the average employee believes that all the content in the organization should be brought together in some form of federated search but now know it isn’t. The result is that they confine themselves to embedded search within specific applications and ignore any option to “search the entire intranet.”

It would be great to see comments from readers who have changed a Web site search experience from a bad scene to one with a positive traffic gain with better search results. Let us know how you did it so we can all learn.

The FAST acquisition of Convera

It has been a couple of weeks since the announcement that Fast Search & Transfer would acquire Convera’s RetrievalWare, a search technology built on the foundation of Excalibur and widely used in government enterprises.

At a recent Boston KM Forum meeting I asked Hadley Reynolds, VP & Director of the Center for Search Innovation at Fast, to comment on the acquisition. He indicated Fast’s interest in building up a stronger presence in the government sector, a difficulty for a Norwegian-based company. I remember Fast as a company launching in the U.S. with great fanfare in 2002 (http://newsbreaks.infotoday.com/nbreader.asp?ArticleID=17223 ) to support FirstGov.gov, a portal to multi-agency content of the U.S. Government. That site has recently been re-launched as http://www.usa.gov/ using the Vivisimo search portal. There must be a story behind the story, as I hope to learn.

To add to the discussion, last week I moderated a session at the Gilbane San Francisco conference at which Helen Mitchell, Senior Search Strategist for Credo Systems and Workgroup Chairperson for the Convera User Group, spoke. I asked Helen before the program about her reaction to the recent announcement. She had already been in contact with Fast and received assurances that Convera Federal Users would be well supported by Fast and they want to actively participate in conversations with the group through on-line and in-person meetings. Helen was positive about the potential for RetrievalWare users gaining from the best of Fast technology while still being supported with the unique capabilities of Convera’s semantic, faceted search.

Erik Schwartz, Director of Product Management from Convera, was also present; I encouraged him and Helen to leverage the RetrievalWare user community to make sure Fast really understands the unique and diverse needs of search within the enterprise. We are all well aware that in the rush to build up large customer bases with a solid revenue stream of maintenance, vendors are likely to sacrifice unique technologies that are highly valued by customers. A bottom-line round of pragmatic cost cutting usually determines what R&D a vendor will fund, foregoing the long term good will that could accrue if they would belly-up to integrating these unique features into their own platform.

Time will tell how serious Fast is in giving its new base a truly valuable customer experience. I would also note that this acquisition has also been observed by a broader information management industry publication, Information Week. See David Gardner’s article at http://www.informationweek.com/news/showArticle.jhtml?articleID=198701793.

Search Help and Usability

Preparing for two upcoming meetings with search themes (Gilbane San Francisco and Boston KM Forum) has brought to mind many issues of search usability. At the core is the issue of search literacy. Offering some fundamental searching tips to non-professional searchers often results in a surprised reaction. (e.g. When told, if seeking information about a specific topic such as "industrial engineering," enclose it in quotes to limit the search to that phrase. Without quotes, you will get all content with “industrial” and “engineering” anywhere in the content with no explicit relationship implied.)

If you are reading this you probably know that, but many do not. In order to learn what people search for on their company intranet and how they type their search requests, I spend time reading search log files. I do this for several reasons:

> To learn terminology searchers are using to guide taxonomy building choices
> To see the way searches are formulated, and followed up
> To inform design decisions about how to make searching easier
> To see what is searched but not found to inform future content inclusion
> To view the searcher’s next step when the results are zero or huge

Two results remain consistent: less than 1% of the searchers place a phrase inside quotations, even when there are multiple words; word are often truncated but do not include a truncation symbol (usually an asterisk, “*”). Both reveal a probable lack of search conventions understanding, a search literacy problem. Here are a couple of possible solutions:
> Put into place better help and training mechanisms to help the lost find their way,
OR
> Remove the legacy practice of forcing command language type symbols on searchers for the most common search requests

Placing punctuation around a search string is a holdover from 30 years ago when searching was done using a command language. Since only a limited number of people ever knew this syntactical format, why does it persist as the default for a phrase search for Web-based search engines?

The solution of providing a better help page and getting people to actually use it is a harder proposition. This one from McGraw-Hill for BusinessWeek Online is pretty simple with just seven tips but who reads it? I expect very few, although it could dramatically improve their search results. http://search.businessweek.com/advanced.jsp.

If you are trying to improve the search experience for your intranet, there are two resources to consult for content usability on all fronts, not just search: useit.com, Jakob Nielsen's Website and Jared Spool’s UIEtips, User Interface Engineering’s free email newsletter. In the meantime, think about whether you need to demand more core search usability or tunable default options from vendors, or whether better interface design could guide searchers to better results.

What is Under the Hood?

Last week I began this entry, re-considered how to make the point and tucked it away. Today I unearthed an article I had not gotten around to putting into my database of interesting and useful citations. Lisa Nadile in The ABCs of Search Engine Marketing, in CIO Magazine, hits the nail on the head with this statement, “Each search engine has its own top-secret algorithm to analyze this data…” This is tongue in cheek so you need to read the whole article to get the humor. Ms. Nadile’s article is geared to Internet marketing but the comments about search engines are just a relevant for enterprise search.

I may be an enterprise search analyst but there are a lot of things I don’t know about the guts of current commercial search tools. Some things I could know if I am willing to spend months studying patents and expensive reports, while other things are protected as trade secrets. I will never know what is under the hood of most products. Thirty years ago I knew a lot about relatively simple concepts like b-tree indexes and hierarchical, relational, networked and associative data structures for products I used and developed.

My focus has shifted to results and usability. My client has to be able to find all the content in their content repository or crawled site. If not, it had better be easy to discover why, and simple to take corrective actions with the search engine’s administration tools, if that is where the problem lies. If the scope of the corpus of content to be searched is likely to grow to hundreds of thousands of documents, I also care about hardware resource requirements and performance (speed) and scalability. And, if you have read previous entries, you already know that I care a lot about service and business relationships with the vendor because that is crucial to long term success. No amount of “whiz bang” technology will overcome a lousy client/vendor relationship.

Finding out what is going on under the hood with some imponderable algorithms isn’t really going to do me or my client any good when evaluating search products. Either the search tool finds stuff the way my client wants to find it, or it doesn’t. “Black art,” trade secret or “patent protected” few of us would really understand the secret sauce anyway.

Trying to Take the High Road

My last blog was in reaction to two recent vendor experiences. One had just briefed me on an enterprise search offering; the other had been ignoring my client’s efforts to get software support, training and respond to bug reports. The second blogged a reaction with a patronizing: “So Lynda should not feel too bad. I know its (sic) frustrating to deal with vendors but not all vendors are the same and she certainly hasn't tried us all.”

With dozens of vendors offering search tools, it was fair to assume that I haven’t tried them all. However, having used search engines of all types since 1974 both as a researcher and analyst I have a pretty good sense of what’s out there. Having evaluated products for clients, and for embedded use in products I brought to market for over 20 years, it doesn’t take me long with a new product to figure out where the problems are. I also talk to a lot of vendors, search users, and read more reports and evaluations than I can count. The evidence about any one product’s strengths and weaknesses piles up pretty quickly. “Searching” for stuff about search has been my career and I do make it my business to keep score on products.

I’m going to continue to hold my counsel on naming different search tools that I’ve experienced for the time being. Instead, in this blog I’ll focus on keeping buyers informed about search technologies in general. My work as a consultant is about helping specific clients look at the best and most appropriate options for the search problems they are trying to solve and to help guide their selection process. Here is some quick generic guidance on making your first search tool choice:

> If you have not previously deployed an enterprise search solution in your domain for the corpus of content you plan to search, do not begin with the highest priced licenses. They are often also the most costly and lengthy implementations and it will take many months to know if a solution will work for you over the long haul.
> Do begin with one or more low cost solutions to learn about search, search product administration, and search engine tuning. This helps you discover what issues and problems are likely to arise, and it will inform you about what to expect (or want) in a more sophisticated solution. You may even discover that a lower cost solution will do just fine for the intended application.
> Do execute hundreds of searches yourself on a corpus of content with which you are very familiar. You want to learn if you can actually find all the content you know is there, and how the results are returned and displayed.
> Do have a variety of types of potential searchers test-drive the installed product over a period of time, review the search logs to get a sense of how they approach searching; then debrief them about their experiences, and whether their search expectations were met.

It is highly unlikely that the first enterprise search product you procure will be the best and final choice. Experience will give you a much better handle on the next selection. It is certainly true that not all vendors or products are the same but you need to do serious reality-based evaluations to learn your most important differentiators.

Setting and Meeting Customer Expectations

I had a briefing from a vendor that is a strong contender for a piece of the enterprise search market this week. The offering is impressive, other reviews have given it high technical marks and the pricing model is reasonable. But because I am currently immersed in the deployment of another enterprise search engine with a client, the issue of vendor client relationship is foremost in my focus.

I asked the CEO of this relatively new offering, what are the fundamental assumptions his company makes about customer technology environments (e.g. the mix of software applications, hardware environment) and the competencies required to integrate his software with that environment. His answer was given strictly in terms of what the IT staff needs to know to bring the product online. My question did have several levels of complexity and was probably badly phrased but I was trying to make a point by asking it.

There are three specific elements missing from search vendors:
> Documentation or explicit models for deployment in environments where there are numerous technological variables to be considered
> Availability of training that takes into consideration the context for enterprise search in a specific customer’s organization
> Frank discussions with customers that set expectations about deployment and implementation, potential bottlenecks, and the need for experienced searchers, search analysts and subject matter experts on the team with the IT group

Downloading software and using automatic installers has become routine; with the launch of a menu and a few simple clicks on boxes on an administrative screen, vendors can claim “out-of-the-box” functionality. Never mind that what you find when you first search your targeted domain is nonsensical, the software finds “stuff.” The IT guys are happy because it was easy to install, met their architecture requirement and, knowing little about the actual corpus of content, they are satisfied that everything works.

I am in a bit of a pickle with the current project, software from another vendor, because:
> What the documentation says will happen when I make certain choices in the set-up does not, in fact, happen when a search is executed
> My attempts through email and phone to schedule training have gone unanswered
> My messages to the support service citing problems also get no response

I’ve only spent two weeks trying to get this software working but three weeks ago, on a holiday, I got a briefing from two executives from this firm because they were “going to be in the area” and wanted face time with a search analyst. Knowing my role as an analyst and as a client you would think they’d answer my phone calls.

What is it that makes the customer experience so easily ignored? All these products look great in demos; what is under the hood is often technologically wonderful but, boy, getting them to work in my environment always seems to be one long nightmare. I wish I could find out what I really need to know. A terrific search engine might help.

Evidence of a Shift Away from Total Enterprise Search

A tough truth about complex and integrated software applications is the lack of expertise and professional depth available to implement and maintain them. This explains a lot about why small business units and project teams often find and deploy their own software tools to get work done.

I am particularly concerned at the lack of will by organizations to fund implementation of applications like enterprise search to aggregate at the retrieval-end the content stored within disparate applications. No rational business planning can justify having workers sift through multiple repositories, each with a separate sign-on, search interface, and search engine protocols just to find a single document. True, organizations need highly competent professionals to meaningfully implement, tune, and administer enterprise search engines. They require the expertise of search analysts, taxonomists, librarians, IT specialists with security, platform, and software development training. However, developing a team of six to twelve “search engineers” to give workers in a thousand person company quick access to relevant content is an ROI no brainer when we know workers waste significant (5 – 15%) amounts of working hours hunting for stuff.

This week’s Information Week article by Nicholas Hoover on Web 2.0 contained a comment about Wells Fargo “…on another Enterprise 2.0 front, integrated search, the company has limited employees’ ability to search across data repositories because of the complex authorization schemes needed to keep people from accessing information they shouldn’t.”

Today’s (Feb. 27th) New York Times headlined with a story about Microsoft buying “a specialized search engine tailored to deliver useful medical information to consumers,” Medstory, Inc. The story goes on to cite comments by Esther Dyson who refers to the technology as “an ontology engine.” This underscores another truth about quality semantic (natural language) search; it depends on the existence of meaningful, topic specific vocabulary and concept maps to work well, a complexity in narrower markets.

Finally, we have seen the recent shift of companies like FAST moving from a strategy of selling solutions directly to enterprises for the purpose of aggregating content through a unified search portal to focusing on niche markets and highly tailored search architectures.

These are just three cases of a shift among search companies to leverage their search technology IP in more lucrative offerings. The losers will be organizations that really do need to deliver content more holistically to workers through a single search engine. Yes, security is a concern, and skilled search technologists must be hired and dedicated to delivering search options that tie directly to business operations.These efforts are not one-off projects but need to be sustained as permanent infrastructure. If you are in a position to influence search procurement solutions make your case for the most suitable software that will really help deliver the best retrieval option company-wide. Be realistic about funding and staffing; then go for it. If Enterprise Search is what you need, make sure that is what you get and deploy.

Which Would You Have? Software as Service or Service with Your Software

I received an unsolicited email from jetBlue yesterday, one of many that I routinely receive from various travel providers. This one was different. I was not one of the thousands stranded by them last week and I have only traveled on jetBlue for one trip. They could have omitted this mea culpa letter to me in hopes that I had not already noticed all the media hype around their operational breakdowns and plans to recover from a faulty infrastructure. However, by calling attention to their lapses in such public ways this week, they have insured that I will include them in future travel planning.

Years ago as the President of a software company, I received a truly disturbing email lashing from a client sent after 6 PM on a Friday. The accusations about my company’s service were vitriolic and uncharacteristic of client reactions. I stayed at the office late gathering all the information I could find from the customer support database to learn what might have precipitated the outburst because I wanted to send a thoughtful, accurate and timely response. Without attacking the client I sent a chronology of inquiries and responses with a copy of a remedy sent to them weeks earlier. Then I went home with hopes that Monday would bring a more constructive dialog between the client and my company. The issues were amicably resolved, the client remained a good client, gave us high marks in referrals, and the matter was never mentioned again.

Unfortunately, personalization of client vendor relationships is missing in too many business relationships. A great amount of marketing copy appears describing how software tools and search interfaces support “personalization.” We know that SaS (software as service) or ASP (application service provider) models have come into their own. We also see the major search software vendors posting record growth and grand projections for even more. What this all adds up to is the convergence of a perfect storm of client disappointment as we experience a total disconnect between what vendors mean by “personalization” and “service,” and what customers want. Customers want software that is intuitively simple to personalize, and service that places the responsibility for software problems squarely with the vendor.

Based on my recent experiences with vendors, I see huge industry problems ahead. These are being exposed at all levels: discussions with sales representatives, exchanges with search company executives, deployment of software issues, documentation and training quality, and exchanges with customer support personnel.

Here is my list of vendor weaknesses:
> Lack of understanding by company representative how their software works
> Failure to really understand prospect needs, environments, and requirements
> Poorly written documentation and training giving no context for how the software might be deployed
> Technically sophisticated features delivered with no coherent path to deployment
> Inability to communicate honestly with clients
> Lack of clarity on what industry standards and terminology mean to clients
> Failure to use their own products by all employees in vendor organizations
> Inattention to building quality support infrastructures to service clients

I am not calling for a “customer bill of rights” for the enterprise search software industry. Instead, I am calling for you who procure software to take control of your own experience by doing a lot more than looking under the hood for technical specifications, features and functionality.

You need to:
> Look inside the vendor’s organization to see what kind of personnel it has, what the turnover is, how many people are supporting service functions compared to developers, etc.
> Listen to what you are being told; do serious validating research, on your own, to discover customers using the software. Talk to as many as you find; look at blogs and chat rooms to discover where the pain points and good experiences lie.
> Read documentation to understand how much time, effort, and expertise the deployment and maintenance will really require.
> Test drive products with your own data.

Every search company can’t grow 100% year-over-year for years on end. You will be suffering mightily for a long time if you make a big investment in one of those who ignore the customer experience. There is also a good chance they’ll be sold off to the lowest bidder once they realize their inability to service their clients and remain profitable. Take your destiny in your own hands; take enterprise search on in slow and measured increments so you will know what you are getting into.

Don’t Skip this Step: Knowing Your Knowledge Assets and How to Find Them

No matter how small your organization or domain, you are going to need tools to find content sooner than you think. Starting with a small amount of content you should already be thinking about what its purpose is, why you would need to find it again and under what circumstances. Trying to retrofit a search strategy and structure to a mountain of disconnected content, is not only very difficult but it is costly. Waiting means that human intelligence, which could have been applied to organizing content well as the supply grew, must be applied later to get it under control. Adding meaningful context around old but valuable content is a very laborious intellectual process.

Growing an organization successfully means tending to not only the products you are creating and selling. It is also about creating an environment in which your growing work force is well supported with a knowledge framework that keeps them centered and confident that content they need to do their jobs can be found quickly, efficiently and accurately.

I am frequently asked by other consultants if I can give advice on how to organize personal files and records. This is hard to answer because my own methods fall short of where I want to be. But in any new project or venture, I try to get a good sense of how content needs to be organized. I do create metadata using a controlled list of terminology. I also have a couple of search tools that I leverage to produce readings for clients on a special topic, or to put my hands on a specific document, article or Web site. Early stage companies need to think about how to safeguard the results of their work and how it will be made accessible to workers on a reliable basis. There are inexpensive search tools that are great for managing small domains. Invest in tools, invest in someone to manage the tools, and plan to continue to invest in the resulting infrastructure of people and tools as the organization’s content and needs grow. Content management and search are overhead expenditures you must make early to prepare for growth and sustainability.

That reminds me, I keeping wondering how many enterprise search vendors use the technologies they build and sell to support their rapidly growing enterprises. That’s a great question to ask your potential search vendor as you decide what tools to procure for your enterprise. Get them to tell you how they use their tools and the benefits they see in their own enterprise. If they aren’t at least using their own search technology in their customer relationship management and technical support knowledge-base operations, think carefully about what that might mean concerning ease of deployment and utilization.

Search Transitions from Support Function to Marketplace Enhancement

My silence last week had more to do with information overload than lack of interesting things to write about. Be forewarned, the floodgates of my brain are beginning to creak open. I just returned from Fast Search’s FastForward 07 conference in San Diego where their current and future visions for search technologies were front and center. While there seem to be no lack of innovations for how to make search engines smarter, faster, and more adaptable, the innovations being hyped at FastForward 07, and by others with only slightly less hyperbole, are notable. Search is becoming sexy and not just for the amount of money that Google and second-ranked Fast are raking in. In this arena search is the new business frontier, the marketplace-enabler, the marketplace-maker.

Consider this, search technologies have been business necessities for 35 years. For the first 30, search was strictly a support feature to many other kinds of finding mechanisms. In the earliest days search was performed by specialists as a service to other operations in the organization. Attempts to market search technology options to line managers, analysts, attorneys and R&D staff were marginal in their success. This is because search was not used enough for these groups to acquire the skill required for it to be really valuable. Once Web search engines exposed everyone to the possibilities of search in a far simpler modality, the innovation light bulbs popped off.

Suddenly search for use within the enterprise has become search for the enterprise’s marketplace, a major business driver that will put an organization’s products, services, and assets squarely in front of the right buying audience. What this means for those poor souls who still need to find the stuff mounting valuelessly in inaccessible silos remains to be seen. I am excited by what I saw but concerned by what I am witnessing. It is great that I may be able to find that weird audio adapter on the Web to let me connect to the sound system in the skating rink. But it is really awful when an engineering firm can’t put it’s hands on the schematic that shows how a circuit board was modified and delivered three years ago to a top customer.

Back to Search Roots for the Enterprise – Structured Search That Is

Structured search (noun) was rooted firmly in the enterprise when publishers of print index resources (e.g. Chemical Abstracts, Index Medicus from the National Library of Medicine, GRA&I from the National Technical Information Service) became available on-line in the early 1970s. The Systems Development Corporation launched ORBIT developed by a team lead by Carlos Cuadra. Orbit was a command driven search tool accessible to professional searchers. In those days searchers were usually special librarians in corporations, large public libraries, government agencies and major universities. Using the ORBIT command language through a terminal connected by a phone line to remote large computers, librarians would type search commands to find data in specific structured fields. These remote computers held electronic versions of paper indices. Citations resulting from a query for specific chemical compounds, diseases, or government reports, would contain information needed to retrieve articles, patents or books from library shelves.

Corporations spent hundreds of thousands of dollars each year to access external specialized, and structured indices, and the journals, conference proceeding, patents and government documents to which the indices pointed. Hard copy (paper or microform) was the only practical way to read content. Computer screens were not accessible to most researchers and even if they had been, content could not be rendered on them in easily readable forms. Also, until computer storage technologies became cheap, indexing large amounts of text (full-text, or unstructured content) was not affordable.

Even with the advent of graphical interfaces, searching for non-specialists made only minor advances in the early-1980s when library systems offered index browsing to find citations. Library users still needed to read content in hard copy. It was only in the late 1980s and early 90s that full-text content began to be searchable by large numbers of library users on CD-ROMs. Users would go to a library computer, which held multiple CD-ROMs containing journals and other subscriptions, and use a menu to find content on the CD-ROMs by typing keywords that would look through all the content to find matches. This was the first routine use of full-text searching by library users.

These technologies are just memories for a few of us, and unknown to most, but they do point to the differentiation between structured and unstructured searching. Both have been around for a couple of decades but it has taken Web search engines to put search in the hands of everyone. Only recently is frustration with retrieving buckets of unfiltered content pushing enterprises to reconfirm the added value of structured searching.

Technical and business users are appreciating the value of being able to search for a precise title, all documents contributed to a specific project, or all presentations delivered by the CEO in the past two years. Each of these searches requires a defined set of data points, stored with the content and retrievable with a search interface that can support the “structured” query.

Yes, librarians have been here before but, just now, the rest of the organization is learning how they managed to get such good search results all along. Structured searching is now a lot simpler than it was in the 1970s. It is only one aspect in enterprise search but it is an important requirement for most enterprise users when they need reliable and clearly defined search results. And, by the way, Carlos is still around building systems for enterprises to manage and search their critical proprietary content.

The Right Message for the Small-Medium Business Market

IBM just launched a very interesting suite of enterprise search products. I have yet to try it or examine the specifications but the marketing message is the right one for the small and medium business enterprise buyer. What I like in the message:
> Pricing ranges from free to reasonable to ? (sky is probably the limit).
> Deployment is simple, intuitive and clean.
> IBM knows that simple and easy is the right call for IT but also the way to keep costs down.
> The solution is scaleable from a departmental solution to the entire business domain on the same basic software platform.

For the short term, I am placing OmniFind on the long list of products to consider for enterprise search. Check out the Web site at:
http://www-306.ibm.com/software/data/enterprise-search/omnifind-enterprise/

How smart is it that IBM and Yahoo have chosen to team in this way? It could be a great strategy.

Disclaimer: This is the first product mention in this very new blog. It won’t be the last; I have a lot of interesting products on my list on which to comment. There will be much backfilling in the next few months but I have to start somewhere. The marketing message resonated; I hope OmniFind users will keep us informed by posting case experiences on whether the product delivers on the promises.

Let’s Not Lose Sight of the Enterprise in Enterprise Search

I seem to be attending a lot of presentations because I think they are going to be about “enterprise search.” Instead they cover a new offering or positioning strategy by a search company seeking to help enterprises monetize their Web sites. There are great business models in this space as Yahoo, Google and Amazon have illustrated. These will morph to offerings, as yet, unimagined. The trouble is that for my audience, I want to help them understand offerings that will help them with searching content already in the enterprise or from outside that they can leverage for business uses: competitive intelligence, product development, supply chain improvements, marketing collateral development. That is what enterprise search is to most people working inside organizations. Admittedly, this is not a new or “sexy” market. I think that vendors of search may be so worn down by how they can make money offering their search tools for searching inside the organization that they may just be talking up other markets to stave off their own boredom with the “inside the enterprise market.” Horizontal markets are tough to deal with (more about that in a later blog entry).

To all you vendors who would like to cut and run, I respectfully ask that you stay. There is a crying need albeit a very big financial challenge in all of this. Enterprises have no idea what their true monetary loses are because workers can’t find “stuff.” There have been plenty of guesses put forth by analysts, but until we see search solutions that don’t take decades (OK years) to implement, who actually knows what gains could be made by really good and easy search tools that find both structured and unstructured content with a minimum of set-up. The pricing models of tools need to make better sense, including ways to chunk the expenditures incrementally with “quick start,” and low overhead options. Why do the search tools with the mightiest claims also come with the highest price tags for licensing but the least out-of-the box functionality?

To all you enterprise information technology seekers of “true enterprise search” you bear responsibility for some of the mess this market is in. You’ve got to write better specifications, learn to start small and demand small to get started, and come to the selection process with real users and professional searchers on the team who will test drive products before they are purchased. If a product doesn’t solve the specific search problem you are trying to solve, don’t buy it. May-be it doesn’t feel like your money when you purchase for your enterprise but it really is your professional responsibility. Would you buy a car that will only take you to the beach or a lake but not to the grocery store?

Good Will and Responsibility

If you signed up for feeds from this site, new posts have been slow coming. Gilbane's announcement of an Enterprise Search Practice has not gone unnoticed. The past two weeks have resulted in more good will than this analyst could easily digest and filter. The good news is that ideas for posting on "enterprise search" are already accumulating faster than they can get written, and the number of enthusiastic well-wishers is encouraging. It looks like we have an audience and community of practice in the making. Thank you to all who have sent their support and good cheer.

Quite a number of responses have come from companies who want to discuss their technology offerings and positioning. At Gilbane we are following up on those requests and beginning to schedule time for discussions and presentations. With the recognition that vendors/suppliers of technologies want ink, and plenty of it, comes a responsibility of which I am acutely aware because I was one of that community for over 20 years. Having founded, in 1980, and lead an integrated library automation firm in the corporate arena, I know how industry press coverage can make or break the fortunes of even the best offerings. While blogs are intended to launch and promote discussions, even play devil's advocate, I don't take this role lightly. Every good intention and hard work by vendors deserves thoughtful and unbiased consideration. It deserves to have analysts who know what they are talking about, and those that would present what they can fairly assess in a useful context. The very definition of analyst (noun) supposes a responsible action, to analyze (verb) the offerings. While my analysis may not focus on what a vendor wants me to consider, it will try to present information that is both helpful and thought-provoking without being mean-spirited or dismissive, and content that helps potential users of the technology focus their own choices and decisions.

Now it's time to get down to business and start making this a more frequent happening. Based on a number of comments, let's begin with clarifying what we mean by enterprise search at Gilbane. While the marketplace often categorizes enterprise search as a specific kind of search product, we at Gilbane don’t. Any technology that serves any type of enterprise by helping it find electronic or physical content through an electronic search interface is fair game. Enterprise search is about looking for content in the organization or for the organization. It may be embedded in a specialized application, may be a platform designed to collectively search and aggregate content from many internal silos, or it may combine search of desktops, enterprise hard drives and the Internet. There is a very big universe of content out there; enterprises need all the search tools they can (afford to) leverage to harvest what they need and when they need it.

Now this analyst’s job is to give you a balance between what the vendors are saying and offering, and what the users really need, and get the two engaging more effectively with each other.

The Enterprise Search Challenge

Enterprise Search has been an illusive dream for too many organizations for too many years. Search technology is ubiquitous but the “holy grail” for most organizations is to be able to find all content within the organization through a single query interface. My instinct is to give a chronology of search over the past four or five decades to guide your understanding of why enterprise search has remained so “out of reach.” I could also describe the ways in which search technologies have evolved and morphed with hundreds of functions and thousands of features. It would certainly help explain why the typical company has a daunting task narrowing its options but it would probably not quicken the selection process.

For now, one view of the current market segmentation is a starting point. Sue Feldman, Research VP, Content Management and Retrieval Solutions at IDC, gave the audience a high level view of the market in a session at Gilbane Boston 2006. She placed enterprise search technology into three big buckets: Appliances and Downloadable Search, Enterprise Search (software) Platforms, and Application Specific Search embedded with other software. She then broadly described the features and functions that characterize each major type. If you have grown up with search in your professional life for over 30 years as I have, it makes perfect sense that this is what we have come to in the market but differentiating the options is a step far less clear-cut.

After the sessions, 15 conference-goers joined me to continue discussing and learning about enterprise search in a roundtable forum. It was hard to know which end of the search animal we should address first to help everyone speak the same language. That is precisely what is making this marketplace such a tough one. Vendors represent a huge variety of solutions, each positioning product(s) for a problem of their definition, offering technology that targets the specific problem. Buyers have multiple search needs but still want a single solution. Further complicating the mix is a dizzying array of search jargon. With vendors and buyers using their own language the market is, frankly, a real mess.

Take Ms. Feldman’s three big buckets and think of one example of search product in each category. Now think about all the types of searches that people in your organization need to perform just to get their routine work done:
> Looking up an address in a directory
> Finding an image for a presentation
> Retrieving a press release your department issued last year on a new product
> Locating a configuration change to a piece of equipment in manufacturing
> and so on…

Can you imagine any single search interface or product from the tools you know that would give you the means to find all of these pieces of information? Can you imagine a single search tool that would answer your query in a couple of simple steps, and able to perform the functions right out of the box? Simple solutions that address the complexity of business variables and technology standards in most organizations make any single solution an unlikely candidate at a reasonable cost.

Blog readers can request answers to questions, ask for help with sorting out the marketplace or definitions to understand the jargon. I invite readers to tell me what you think needs to be talked about and I’ll give it my best shot. What do you need to know first to tread through the search marketplace?

Searching for Enterprise Search

The Gilbane Group has invited me to lead a new practice area for the Gilbane community. Based on my recent experience at Gilbane Boston, November 2006, there is the foundation for a substantial community of practice around the topic of Enterprise Search. When I invited attendees of conference sessions on search to a preliminary roundtable discussion about enterprise search, over 15 people signed up. About 12 people dropped in for all or part of what turned out to be a two hour lunch break with a fluid conversation about what search means, what users are seeking for their organizations and what vendors (service and product) have to offer.

I am inviting anyone who drops in on this blog to continue the conversation. By sharing needs, product offerings, definitions, problems and case studies, participants in this blog have the opportunity to contribute to a community of practice among a highly diverse audience of professionals concerned with this topic of search. We will learn what is working or not, what tools, tips and processes have been used and leveraged for improving business performance in any type of organization.

You will note a group of categories that I established because I have something I would like to share sooner rather than later about these topics. Not everything I may have to say in the next few weeks or months will fit neatly; expansion is inevitable. The categories are broad until we begin to accumulate content in other areas. One thing is certain; technology changes and changes our thinking. A year from now it will be interesting to look back at recommendations, advice, enthusiasms and endorsements in this time period and see where reversals happen and attitudes morph. We are all capable of love hate relationships with technology and I am as fickle as the next person.

Stay tuned and see where we are January 1, 2008. Still blogging, I hope.

Enterprise Search Practice Blog

Analysis, opinion, and advice on enterprise search technologies and applications

Posts by Analyst

Categories

Search

Archives

Gilbane Group Links

Lynda Moulton's Entries

May 22, 2008

Search your Enterprise Really Needs

May 15, 2008

Interview with Steve Arnold about "Beyond Search" with Jess Bratcher

Enterprise Search and the Conference Season

April 14, 2008

Only Humans can Ensure the Value of Search in Your Enterprise

April 3, 2008

Parsing the Enterprise Search Landscape

March 14, 2008

Enterprise Search Adopters

March 2, 2008

Ontologies and Semantic Search

February 22, 2008

Taxonomy and Enterprise Search

February 12, 2008

Search Engines Under the Hood

February 1, 2008

Search Behind the Firewall aka Enterprise Search

January 24, 2008

Search Adoption is a Tricky Business: Knowledge Needed

January 16, 2008

Nothing Like a Move by Microsoft to Stir up Analysis and Expectations

January 8, 2008

A Call for Papers and Microsoft creates a FAST Opening in the New Year

December 31, 2007

Enterprise Search and Its Semantic Evolution

December 14, 2007

Enterprise Search: Leveraging and Learning From Web Search and Content Tools

December 6, 2007

Case Studies: Enterprise Search Success Stories

November 27, 2007

Leading Enterprise Initiatives or Reacting to Crisis

November 14, 2007

Surrounding and Supporting Enterprise Search

October 30, 2007

Gearing up for Gilbane Boston 2007

October 19, 2007

Eating Your Own Dog Food

October 12, 2007

Collaboration and Expertise Bring Focus to Enterprise Search

October 5, 2007

Trust is a Pact between a Vendor and Its Customer

September 21, 2007

About Analysts, Other Naysayers and Search

September 4, 2007

Using Expertise and Enterprise Search to Drive New Knowledge

August 27, 2007

Mobile Search, Etc. and Early Adoption

August 16, 2007

Sharepoint and Search

July 31, 2007

The Marginal Influence of E-commerce Search and Taxonomies on Enterprise Search Technologies

July 20, 2007

Search and Need

July 13, 2007

Random Notes from the World on Search

June 22, 2007

Respect for Complexity and Security are Winners

June 15, 2007

Turbo Search Engines in Cars; it is not the whole solution.

June 7, 2007

A Story About Structured Content and Search

May 30, 2007

Reflecting on BI Shifts and Google Big Moves in the Search Market

May 21, 2007

Will Steve Arnold Scare IT Into Taking Search in the Enterprise Seriously?

May 16, 2007

Mapping Search Requirements

May 10, 2007

Will Search Technology Ever Become a Commodity?

April 25, 2007