The focus is on an enterprise with a substantive amount of content aggregated from a diverse universe of industry specific information, and what to do about searching it. If the information has been parsed and stored in an RDBMS database, is it not better to leverage the SQL query engine native to the RDBMS? Typical database engines might be: DB2, MS Access, MS SQL, MySQL, Oracle or Progress Software.

To be clear, I am not a developer but worked closely with software engineers for 20 years when I owned a software company. We worked with several DBMS products, three of them supported SQL queries and the application we invented and supported was a forerunner of today's content management systems with a variety of retrieval (search) interfaces. The retrievable content our product supported was limited to metadata plus abstracts up to two or three pages in length; the typical database sizes of our customers ranged from 250,000 to a couple of million records.

This is small potatoes compared to what search engines typically traverse and index today but scale was always an issue and we were well aware of the limitations of the SQL engines to support contextual searching, phrase searching and complex Boolean queries. It was essential that indexes be built in real time, when records were added whether manually through screen forms, or through batch loads. The engine needed to support explicit adjacency (phrase) searching as well as key words anywhere in a field, in a record, or in a set. Saving and re-purposing results, storing search strategies, narrowing large sets incrementally, and browsing indexes of terminology (taxonomy navigation) to select unique terms that would enable a Boolean "and" or "or" query were part of the application. When our original text-based DBMS vendor went belly-up, we spent a couple of years test driving numerous RDBMS products to find one that would support the types of searches our customers expected. We settled on Progress Software primarily because of its support for search and experience as an OEM to application software vendors, like us. Development time was minimized because of good application building tools and index building utilities.

So, what does that have to do with the original question, native RDBMS search vs. standalone enterprise search? Based on discussions and observations with developers trying to optimize search for special applications, using generic search tools for database retrieval, I would make the following observations. Search is very hard and advanced search, including concept searching, Boolean operations, and text analytics, is harder still. Developers of enterprise search solutions have grappled with and solved search problems that need to be supported in environments where content is dynamically changing and growing, different user interfaces for diverse audiences and types of queries are needed, and query results require varieties of display formats. Also, in e-commerce applications, interfaces require routine screen face lifts that are best supported by specialized tools for that purpose.

Then you need to consider all these development requirements; they do not come out-of-the-box with SQL search:

• Full text indexes and database field or metadata indexes require independent development efforts for each database application that needs to be queried.


• Security databases must be developed to match each application where individual access to specific database elements (records or rows) is required.


• Natural language queries require integration with taxonomies, thesauri, or ontologies; this means software development independent of the native search tools.


• Interfaces must be developed for search engine administrators to make routine updates to taxonomies and thesauri, retrieval and results ranking algorithms, adjustments to include/exclude target content in the databases. These content management tasks require substantive content knowledge but should not require programming expertise and must be very efficient to execute.


• Social features that support interaction among users and personalization options must be built.


• Connectors need to be built to federate search across other content repositories that are non-native and may even be outside the enterprise.

We have seen companies like Exalead, ISYS, and Vivisimo gain on former leaders. Microsoft is often listed as an industry leader because it acquired former leader FAST while companies with solid products for verticals, like Recommind in law and financial services, are often overlooked because they lack the total company revenues of a Microsoft that sells a lot more software than enterprise search.

This past week two industry news items caused me to reflect on the potential impact of announcements that, while not surprising, can upset the plans of buyers of search technology. The first was the announcement that Autonomy is planning to procure Interwoven. That Interwoven is being acquired is no surprise, since the company was being groomed for acquisition. However, this appears to be the first instance of a "search" company acquiring a "content management/document management" company. The norm has been that search companies get bought to fill a need by ECM or CMS vendors. For anyone planning to procure Interwoven because of its embedded Vivisimo Velocity for Universal search in its Worksite product, this does put a wrinkle in the fabric. What a shame because it is going to be a while before the actual impact is really known and could slow sales. The cost to buyers having to accept Autonomy's IDOL instead of Velocity could be significant. The effect could be on both licensing and deployment because Velocity has been an efficient install for most enterprises. Autonomy has got a big ramp up to shift from being a search company to becoming an ECM supplier and some will take a wait and see attitude, regardless of the Idol reputation.

The second big announcement, of course, is the departure from Microsoft of John Marcus Lervik, a co-founder of FAST and recently named Executive VP in a newly created position for Enterprise Search at Microsoft. I'm sure you'll be seeing plenty about the reasons elsewhere. However, the difficulty for those buyers who are depending on FAST's search technology to be integrated sooner rather than later in Microsoft's offerings has just been made more complicated as one of the original leaders of FAST is leaving the team.

Two years ago I commented to FAST executives about the need for vendors on a rapid growth path to make the buying, business and support experience for customers a priority, beyond technology enhancements; so, I take little consolation in seeing this turmoil. If you are a buyer, take a good hard look behind the technology to see what else you will be dealing with as you make plans to acquire software.

A recent examination of those early decisions showed me where there are gaps in content, perhaps reflecting that some of those topics were:

• Not so important


• Not currently in my thinking about the industry


• OR Not well defined

I also know that on several occasions I couldn't find a good category in my list for a blog I had just written. Being a former indexer and heavy user of controlled vocabularies, on most occasions I resisted the urge to create a new category and found instead the "best fit" for my entry. I know that when the corpus of content or domain is small, too many categories are useless for the reader. But now, as I approach 100 entries, it is time to reconsider where I want to go with blogging about enterprise search.

In the short term, I am going to try to provide entries for scantily covered topics because I still think they are all relevant. I'll probably add a few more along the way or perhaps make some topics a little more granular.

Taxonomies are never static, and require periodic review, even when the amount of content is small. Taxonomists need to keep pace with current use of terminology and target audience interests. New jargon creeps in although I prefer to use generic and terms broadly understood in the technology and business world.

That gives you an idea of some of my own taxonomy process. To add to the entries on terminology (definitions) and taxonomies, I am posting a glossary I wrote for last year's report on the enterprise search market and recently updated for the Gilbane Workshop on taxonomies. While the definitions were all crafted by me, they are validated through the heavy use of the Google "define" feature. If you aren't already a user, you will find it highly useful when trying to pin down a definition. At the Google search box, simply type define: xxx xxx (where xxx represents a word or phrase for which you seek a definition). Google returns all the public definition entries it finds on the Internet. My definitions are then refined based on what I learn from a variety of sources I discover using this technique. It's a great way to build your knowledge-base and discover new meanings.

Glossary Taxonomy and Search-012009.pdf

Search appliances are sold with licenses that imply some high level of performance without a lot of support, while open source search tools are downloadable for free. As speakers about both open source and appliances made perfectly clear at our recent Gilbane Conference, both come with requirements for human support. When any enterprise search product or tool is selected and procured, there is a presumed business case for acquisition. What acquirers need to understand above all else is the cost of ownership to achieve the expected value. This means people and people with expertise on an ongoing basis.

Particularly when budgets are tight and organizations lay off workers, we discover that those with specialized skills and expertise are often the first to go. The jack-of-all-trades, or those with competencies in maintaining ubiquitous applications are retained to be "plugged in" wherever needed. So, where does this leave you for support of the search appliance that was presumed to be 100% self-maintaining, or the open source code that still needs bug fixes, API development and interface design-work?

This is the time to look to system integrators and service companies with specialists in tools you use. They are immersed in the working innards of these products and will give you better support through service contracts, subscriptions or labor-based hourly or project charges than you would have received from your in-house generalists, anyway.

You may not see specialized system houses or service companies listed by financial publications as a growth business, but I am going to put my confidence in the industry to spawn a whole new category of search service organizations in the short term. Just-in-time development for you and lower overhead for your enterprise will be a growing swell in 2009. This is how outsourcing can really bring benefits to your organization.

Post-post note - Here is a related review on the state-of-open source in the enterprise: The Open Source Enterprise; its time has come, by Charles Babcock in Information Week, Nov. 17, 2008. Be sure to read the comments, too.

We do not hide the fact that, as industry analysts, we also consult to various content software companies. When doing so, I try to keep in mind that the market will be served best when I honestly advocate for software and service improvements that will benefit buyers. This is a value to those who sell and those who buy software. My consulting to vendors indirectly benefits both audiences.

Analysts also consult to buyers, to help them make informed decisions about technology decisions and business relationships. I particularly enjoy and value those experiences because what I learn about enterprise buyers' needs and expectations can translate directly into advice to vendors. This is an honest brokering role that comes naturally because I have been a software vendor and also in a position to make many software procurement decisions, particularly tools and applications that were used by my development and service teams. I'm always enthusiastic to be in a position to share important information about products with buyers and information about buying audiences with those who build products. This can be done effectively while preserving confidentiality on both sides and making sure that everyone gets something out of the communications.

As an analyst, I receive a lot of requests by vendors to listen to, by phone and Web, briefings on their products, or to meet, one-on-one with their executives. You may have noticed that I don't write reviews of specific products although, in a particular context, I may reference products and applications. While we understand the reason that product vendors want analysts to pay attention to them, I don't find briefings particularly enlightening unless I know nothing about a company and its offerings. For these types of overviews, I can usually find what I want to know on their Web site, in press releases and by poking around the Web. During briefings I want to drive the conversation toward user experiences and needs.

What I do like to do is talk to product users about their experiences with a vendor or a product. I like to know what the implementation and adoption experience is like and how their organization had been affected by product use, both benefits and drawbacks. It is not always easy to gain access to customers but I have ways of finding them and also encourage readers of this blog to reach out with your stories. I am delighted to learn more through comments to the blog, an email or phone call. If you are willing to chat with me for a while, I will call you at your convenience.

The original topic I planned to write about this week will have to wait because, after receiving over 20 invitations to "be briefed" in the past few days, I decided it was more important to let readers know who I want to be briefed by - search technology users are my number one target. Vendors please push your customers in this direction if you want me to pay attention. This can bring you a lot of value, too. It is a matter of trust.

The weak link in the search market is product selection methods. My first thought is that we are in a state of technological riches without clear guideposts for which search models work best in any given enterprise. Those tasked to select and purchase products are not well-educated about the marketplace but are usually not given budget or latitude to purchase expert analysis when it is available. It is a sad commentary to view how organizations grant travel budgets to attend conferences where only limited information can be gathered about products but will not spend a few hundred dollars on in-depth comparative expert analyses of a large array of products.

My sources for this observation are numerous, confirmed by speakers in our Gilbane conference search track sessions in Boston and San Francisco. As they related their personal case histories for selecting products, speakers shared no tales of actually doing literature searches or in-depth research using resources with a cost associated. This underscores another observation, those procuring search do not know how to search and operate in the belief that they can find "good enough" information using only "free stuff." Even their review of material gathered is limited to skimming rather than a systematic reading for concrete facts. This does not make for well-reasoned selections. As noted in an earlier entry, a widely published chart stating that product X is a leader does nothing to enlighten your enterprise's search for search. In one case, product leadership is determined primarily by the total software sales for the "leader" of which search is a miniscule portion.

Don't expect satisfaction with search products to rise until buyers develop smarter methods for selection and better criteria for making a buy decision that suits a particular business need.

Random Thoughts. It will be a very long time before we see a universally useful, generic search function embedded in Microsoft (MS) product suites as a result of the FAST acquisition. Asked earlier in the year by a major news organization whether I though MS had paid too much for FAST, I responded "no" if what they wanted was market recognition but "yes" if they thought they were getting state-of-the-art-technology. My position holds; the financial and legal mess in Norway only complicates the road to meshing search technology from FAST with Microsoft customer needs.

I've wondered what has happened to the OmniFind suite of search offerings from IBM. One source tells me it makes IBM money because none of the various search products in the line-up are standalone, nor do they provide an easy transition path from one level of product to another for upward scaling and enhancements. IBM can embed any search product with any bundled platform of other options and charge for lots of services to bring it on-line with heavy customization.

Three platform vendors seem to be penetrating the market slowly but steadily by offering more cohesive solutions to retrieval. Native search solutions are bundled with complete content capture, publishing and search suites, purposed for various vertical and horizontal applications. These are Oracle, EMC, and OpenText. None of these are out-of-the-box offerings and their approach tends to appeal to larger organizations with staff for administration. At least they recognize the scope and scale of enterprise content and search demands, and customer needs.

On User Presentations at the Boston Gilbane Conference, I was very pleased with all sessions, the work and thought the speakers put into their talks. There were some noteworthy comments in those on Semantic Search and Text Technologies, Open Source and Search Appliances.

On the topic of semantic (contextual query and retrieval) search, text mining and analytics, the speakers covered the range of complexities in text retrieval, leaving the audience with a better understanding of how diverse this domain has become. Different software application solutions need to be employed based on point business problems to be solved. This will not change, and enterprises will need to discriminate about which aspects of their businesses need some form of semantically enabled retrieval and then match expectations to offerings. Large organizations will procure a number of solutions, all worthy and useful. Jeff Catlin of Lexalytics gave a clear set of definitions within this discipline, industry analyst Curt Monash provoked us with where to set expectations for various applications, and Win Carus of Information Extraction Systems illustrated the tasks extraction tools can perform to find meaning in a heap of content. The story has yet to be written on how semantic search is and will impact our use of information within organizations.

Leslie Owens of Forrester and Sid Probstein of Attivio helped to ground the discussion of when and why open source software is appropriate. The major take-way for me was an understanding of the type of organization that benefits most as a contributor and user of open source software. Simply put, you need to be heavily vested and engaged on the technical side to get out of open source what you need, to mold it to your purpose. If you do not have the developers to tackle coding, or the desire to share in a community of development, your enterprise's expectations will not be met and disappointment is sure to follow.

Finally, several lively discussions about search appliance adoption and application (Google Search Appliance and Thunderstone) strengthen my case for doing homework and making expenditures on careful evaluations before jumping into procurement. While all the speakers seem to be making positive headway with their selected solutions, the path to success has involved more diversions and changes of course than necessary for some because the vetting and selecting process was too "quick and dirty" or dependent on too few information sources. This was revealed: true plug and play is an appliance myth.

What will 2009 bring? I'm looking forward to seeing more applications of products that interest me from companies that have impressed me with thoughtful and realistic approaches to their customers and target audiences. Here is an uncommon clustering of search products.

Multi-repository search across database applications, content collaboration stores document management systems and file shares: Coveo, Autonomy, Dieselpoint, dtSearch, Endeca, Exalead, Funnelback, Intellisearch, ISYS, Oracle, Polyspot, Recommind, Thunderstone, Vivisimo, and X1. In this list is something for every type of enterprise and budget.

Business and analytics focused software with intelligence gathering search: Attensity, Attivio, Basis Technology, ChartSearch, Lexalytics, SAS, and Temis.

Comprehensive solutions for capture, storage, metadata management and search for high quality management of content for targeted audiences: Access Innovations, Cuadra Associates, Inmagic, InQuira, Knova, Nstein, OpenText, ZyLAB.

Search engines with advanced semantic processing or natural language processing for high quality, contextually relevant retrieval when quantity of content makes human metadata indexing prohibitive: Cognition Technologies, Connotate, Expert System, Linguamatics, Semantra, and Sinequa

Content Classifier, thesaurus management, metadata server products have interplay with other search engines and a few have impressed me with their vision and thoughtful approach to the technologies: MarkLogic, MultiTes, Nstein, Schemalogic, Seaglex, and Siderean.

Search with a principal focus on SharePoint repositories: BA-Insight, Interse, Kroll Ontrack, and SurfRay.

Finally, some unique search applications are making serious inroads. These include Documill for visual and image, Eyealike for image and people, Krugle for source code, and Paglo for IT infrastructure search.

This is the list of companies that interest me because I think they are on track to provide good value and technology, many still small but with promise. As always, the proof will be in how they grow and how well they treat their customers.

That's it for a wrap on Year 2 of the Enterprise Search Practice at the Gilbane Group. Check out our search studies at http://gilbane.com/Research-Reports.html and PLEASE let me hear your thoughts on my thoughts or any other search related topic via the contact information at http://gilbane.com/contact.html

EST-1: Plug-and Play: Enterprise Experiences with Search Appliances

• So you want to implement an enterprise search solution? Speaker: Angela A. Foster, FedEx Services, FedEx.com Development, and Dennis Shirokov, Marketing Manager, FedEx Digital Access Marketing.


• The Make or Buy Decision at the U.S. General Services Admin. Speaker: Thomas Schaefer, Systems Analyst and Consultant, U.S. General Services Administration


• Process and Architecture for Implementing GSA at MITRE. Robert Joachim, Info Systems Engr, Lead, The MITRE Corporation.

EST-2: Search in the Enterprise When SharePoint is in the Mix

• Enterprise Report Management: Bringing High Value Content into the Flow of Business Action. Speaker: Ajay Kapur, VP of Product Development, Apps Associates


• Content Supply? Meet Knowledge Demand: Coveo SharePoint integration. Speaker: Marc Solomon, Knowledge Planner, PRTM.


• In Search of the Perfect Search: Google Search on the Intranet. Speaker: June Nugent, Director of Corporate Knowledge Resources, NetScout Systems,

EST-3: Open Source Search Applied in the Enterprise

• Context for Open Source Implementations. Speaker: Leslie Owen, Analyst, Forrester Research


• Intelligent Integration: Combining Search and BI Capabilities for Unified Information Access. Speaker: Sid Probstien, CTO, Attivio

EST-4: Search Systems: Care and Feeding for Optimal Results

• Getting Off to a Strong Start with Your Search Taxonomy. Speaker: Heather Hedden, Principal Hedden Information Management


• Getting the Puzzle Pieces to Fit; Finding the Right Search Solution(s) Patricia Eagan, Sr. Mgr, Web Communications, The Jackson Laboratory.


• How Organizations Need to Think About Search. Speaker: Rob Wiesenberg, President & Founder, Contegra Systems

EST-5: Text Analytics/Semantic Search: Parsing the Language

• Overview and Differentiators: Text Analytics, Text Mining and Semantic Technologies. Jeff Catlin, CEO, Lexalytics


• Reality and Hype in the Text Retrieval Market. Curt Monash, President, Monash Research.


• Two Linguistic Approaches to Search: Natural Language Processing and Concept Extraction. Speaker: Win Carus, President and Founder, Information Extraction Systems

Exhibitors with a Search Focus:
Attivio Active Intelligence
Coveo G2B
Exalead Cloud View
IntelliSearch
Lexalytics and infonic Company

Dissatisfaction comes from the numbers of these interfaces and the lack of cohesive roadmap to all there is to be found. You already know what you know and what you need to know. Sometimes you know how to find what you need to know but more often you don't know and stumble through a variety of possibilities up to and including asking someone else how to find it. That missing roadmap is more than an annoyance; it is a major encumbrance to doing your job and top management does not get it. They simply won't accept that one or two content roadmap experts (overhead) could be saving many people-years of company time and lost productivity.

In most cases, the simple notion of creating clear guidelines and signposts to enterprise content is a funding showstopper. It takes human intelligence to design and build that roadmap and put the technology aids in place to reveal it. Management will fund technology but not the content architects, knowledge "mappers" and ongoing gatekeepers to stay on top of organizational change, expansions, contractions, mergers, rule changes and program activities that evolve and shift perpetually. They don't want infrastructure overhead whose primary focus, day-in and day-out, will be observing, monitoring, communicating, and thinking about how to serve up the information that other workers need to do their jobs. These people need to be in place as the "black-boxes" that keep search tools in tip-top operating form.

Last week I commented on the products that will be featured in the Search Track at Gilbane Boston, Dec. 3rd and 4th. What you will learn about these tools is going to be couched in case studies that reveal the ways in which search technology is leveraged by people who think a lot about what needs to be found and how search needs to work in their enterprises. They will talk about what tools they use, why and what they are doing to get search to do its job. I've asked the speakers to tell their stories and based on my conversations with them in the past week, that is what we will hear, the reality!

From search appliances Thunderstone and Google Search Appliance, to platform search solutions based on Oracle Secure Enterprise Search, and standalone search products Coveo, Exalead, and ISYS, we will hear from those who have been involved in selecting, implementing and deploying these solutions for enterprise use. From a Forrester industry analyst and Attivio developer we'll hear about open source options and how they are influencing enterprise search development. The search sessions will be rounded out as we explore the influences and mergers of text mining, text analytics with Monash Research and semantic technologies (Lexalytics and InfoExtract) as they relate to other enterprise search options. There will be something for everyone in the sessions and in the exhibit hall.

Personally, I am hoping to see many in the audience who also have search stories within their own enterprises. Those who know me will attest to my strong belief in communities of practice and sharing. It strengthens the marketplace place when people from different types of organizations share their experiences trying to solve similar problems with different products. Revealing competitive differentiators among the numerous search products is something that pushes technology envelopes and makes for a more robust marketplace. Encouraging dialogue about products and in-the-field experiences is a priority for all sessions at the Gilbane Conference and I'll be there to prompt discussion for all five search sessions. I hope you'll join me in Boston.

Because I follow enterprise search for the Gilbane Group while maintaining a separate consulting practice in knowledge management, I am struggling with his conflation of the two terms or even the migration of one to the other. The search we talk about is a set of software technologies that retrieve content. I'm tired of the debate about the terminology "enterprise search" vs. "behind the firewall search." I tell vendors and buyers that my focus is on software products supporting search executed within (or from outside looking in) the enterprise on content that originates from within the enterprise or that is collected by the enterprise. I don't judge whether the product is for an exclusive domain, content type or audience, or whether it is deployed with the "intent" of finding and retrieving every last scrap of content lying around the enterprise. It never does nor will do the latter but if that is what an enterprise aspires to, theirs is a judgment call I might help them re-evaluate in consultation.

It is pretty clear that Mr. Altendorf is impressed with the potential for Fast and Microsoft so he knows they are firmly entrenched in the software business. But knowledge management (KM) is not now, nor has it ever been, a software product or even a suite of products. I will acknowledge that KM is a messy thing to talk about and the label means many things even to those of us who focus on it as a practice area. It clearly got derailed as a useful "discipline" of focus in the 90s when tool vendors decided to place their products into a new category called "knowledge management."

It sounded so promising and useful, this idea of KM software that could just suck the brains out of experts and the business know-how of enterprises out of hidden and lurking content. We know better, we who try to refine the art of leveraging knowledge by assisting our clients with blending people and technology to establish workable business practices around knowledge assets. We bring together IT, business managers, librarians, content managers, taxonomists, archivists, and records managers to facilitate good communication among many types of stakeholders. We work to define how to apply behavioral business practices and tools to business problems. Understanding how a software product is helpful in processes, its potential applications, or to encourage usability standards are part of the knowledge manager's toolkit. It is quite an art, the KM process of bringing tools together with knowledge assets (people and content) into a productive balance.

Search is one of the tools that can facilitate leveraging knowledge assets and help us find the experts who might share some "how-to" knowledge, but it is not, nor will it ever be a substitute for KM. You can check out these links to see how others line up on the definitions of KM: CIO introduction to KM and Wikipedia. Let's not have the "KM is dead" discussion again!

Most sizable non-profits want their collections of content and other information assets exposed to the public. But each department manages its own content collections with software that is unique to their specific professional methods and practices. In the corporate world the mix will include human resources (HR), enterprise resource management (ERP) systems, customer relationship management (CRM), R & D document management systems and collaboration tools. Many corporations have or "had" library systems that reflected a mix of internally published reports and scholarly collections that support R & D and special areas such as competitive intelligence. Corporations struggle constantly with federating all this content in a single search system.

Non-profit organizations have similar disparate systems constructed for their special domain, museums or research institutions. One area that is similar between the corporate and non-profit sector is libraries, operating with software whose interfaces hearken back to designs of the late 1980s or 90s. Another by-product of that era was the catalog record in a format devised by the Library of Congress for the electronic exchange of records between library systems. It was never intended to be the format for retrieval. It is similar to the metadata in content management systems but is an order of magnitude more complex and arcane to the typical person doing searching. Only librarians and scholars really understand the most effective ways to search most library systems; therein lies the "public access" problem. In a corporation a librarian often does the searching.

However, a visitor to a museum Web site would expect to quickly find a topic for which the museum has exhibit materials, printed literature and other media, all together. This calls for nomenclature that is "public friendly" and reflects the basic "aboutness" of all the materials in museum departments and collections. It is a problem when each library and curatorial department uses a different method of categorizing. Libraries typically use Library of Congress Subject Headings. What makes this problematic is that topics are so numerous. The number of possible subject headings is designed for the entire population of all Library of Congress holdings, not a special collection of a few tens of thousands of materials. Almost no library systems search for words "contained in" the subject headings if you try to browse just the Subject index. If I am searching Subjects for all power generation materials and a heading such as electric power generation is used, it will not be found because the look-up mechanism only looks for headings that "begin with" power generation.

Let's cut to the chase; mountains of metadata in the form of library cataloging are locked inside library systems within non-profit institutions. It is not being searched at the search box when you go to a museum Web site because it is not accessible to most "enterprise" or "web site" search engines. Therefore, a separate search must be done in the library system using a more complex approach to be truly thorough.

We have a big problem if we are to somehow elevate library collections to the same level of importance as the rest of a museum's collections and integrate the two. Bigger still is the challenge of getting everything indexed with a normalized vocabulary for the comfort of all audiences. This is something that takes thought and coordination among professionals of diverse competencies. It will not be solved easily but it must be done for institutions to thrive and satisfy all their constituents. Here we have yet another example of where enterprise search will fail to satisfy, not because the search engine is broken but because the underlying data is inappropriately packaged for indexes to work as expected. Yet again, we come to the realization that we need people to recognize and fix the problem.

I bring this up because the latest Gartner "magic quadrant" lists Microsoft (MS) as the "leader" in enterprise search; this is the same place Gartner has positioned Fast Search & Transfer in the past. Whether this is because Fast's assets are now owned by MS or because Gartner really believes that Microsoft is the leader, I still beg to strongly differ.

I have been perplexed by the Microsoft/Fast deal since it was announced earlier this year because, although Fast has always offered a lot of search technology, I never found it to be a compelling solutions for any of my clients. Putting aside the huge upfront capital cost for licenses, the staggering amount of development work, and time to deployment there were other concerns. I sensed a questionable commitment to an on-going, sustainable, unified and consistent product vision with supporting services. I felt that any client of mine would need very deep pockets indeed to really make a solid value case for Fast. Most of my clients are already burned out on really big enterprise deployments of applications in the ERP and CRM space, and understand the wisdom of beginning with smaller value-achievable, short-term projects on which they can build.

Products that impress me as having much more "out-of-the-box" at a more reasonable cost are clearly leaders in their unique domains. They have important clients achieving a good deal of benefit at a reasonable cost, in a short period of time. They have products that can be installed, implemented and maintained internally without a large staff of administrators, and they have good reputations among their clients for responsiveness and a cohesive series of roll-outs. Several have as many or more clients than Fast ever had (if we ever know the real number). Coveo, Exalead, ISYS, Recommind, Vivisimo, and X1 are a few of a select group that are marking a mark in their respective niches, as products ready for action with a short implementation cycle (weeks or months not years).

Autonomy and Endeca continue to bring value to very large projects in large companies but are not plug-and-play solutions, by any means. Oracle, IBM, and Microsoft offer search solutions of a very different type with a heavy vendor or third-party service requirement. Google Search Appliance has a much larger installed base than any of these but needs serious tuning and customization to make it suitable to enterprise needs. Take the "leadership" designation with a big grain of salt because what leads on the charts may be exactly what bogs you down. There are no generic, one-suit-fits-all enterprise search solutions including those in the "leaders" quadrant.

I'll try to share a clearer view of the model and explain why new systems of organizing content in enterprise search are quite different than the decimal model.

Classification is a good generic term for defining physical organizing systems. Unique animals and plants are distinguished by a single classification in the biological naming system. So too are books in a library. There are two principal classification systems for arranging books on the shelf in Western libraries: Dewey Decimal and Library of Congress (LC). They each use coding (numeric for Dewey decimal and alpha-numeric for Library of Congress) to establish where a book belongs logically on a shelf, relative to other books in the collection, according to the book's most prominent content topic. A book on nutrition for better health might be given a classification number for some aspect of nutrition or one for a health topic, but a human being has to make a judgment which topic the book is most "about" because the book can only live in one section of the collection. It is probably worth mentioning that the Dewey and LC systems are both hierarchical but with different priorities. (e.g. Dewey puts broad topics like Religion and Philosophy and Psychology at top levels and LC puts those two topics together while including more scientific and technical topics at the top of the list, like Agriculture and Military Science.)

So why classify books to reside in topic order? It requires a lot of labor to move the collections around to make space for new books. It is for the benefit of the users, to enable "browsing" through the collection, although it may be hard to accept that the term browsing was a staple of library science decades before the internet. Library leaders established eons ago the need for a system of physical organization to help readers peruse the book collection by topic, leading from the general to the specific.

You might ask what kind of help that was for finding the book on nutrition that was classified under "health science." This is where another system, largely hidden from the public or often made annoyingly inaccessible, comes in. It is a system of categorization in which any content, book or otherwise, can be assigned an unlimited number of categories. Wondering through the stacks, one would never suspect this secret way of finding a nugget in a book about your favorite hobby if that book was classified to live elsewhere. The standard lists of terms for further describing books by multiple headings are called "subject headings" and you had to use a library catalog to find them. Unfortunately, they contain mysterious conventions called "sub-divisions," designed to pre-coordinate any topic with other generic topics (e.g. Handbooks, etc. and United States). Today we would call these generic subdivision terms, facets. One reflects a kind of book and the other reveals a geographical scope covered by the book.

With the marvel of the Web page, hyperlinking, and "clicking through" hierarchical lists of topics we can click a mouse to narrow a search for handbooks on nutrition in the United States for better health beginning at any facet or topic and still come up with the book that meets all four criteria. We no longer have to be constrained by the Dewey model of browsing the physical location of our favorite topics, probably missing a lot of good stuff. But then we never did. The subject card catalog gave us a tool for finding more than we would by classification code alone. But even that was a lot more tedious than navigating easily through a hierarchy of subject headings, narrowing the results by facets on a browser tab and further narrowing the results by yet another topical term until we find just the right piece of content.

Taking the next leap we have natural language processing (NLP) that will answer the question, "Where do I find handbooks on nutrition in the United States for better health?" And that is the Holy Grail for search technology - and a long way from Mr. Dewey's idea for browsing the collection.

We entered the Web search era bundling content into a directory structure, like the "Yellow Pages," or organizing query results into "folders" labeled with broad topics. The controlled vocabulary that represented directory topics or folder labels became known as a taxonomic structure, with the early ones at NorthernLight and Yahoo crafted by experts with knowledge of the rules of controlled vocabulary, thesaurus development and maintenance. Google derailed that search model with its simple "search box" requiring only a word or phrase to grab heaps of results. Today we are in a new era. Some people like searching by typing keywords in a box, while others prefer the suggestions of a directory or tree structure. Building taxonomic structures for more than e-commerce sites is now serious business for searches within enterprises where many employees prefer to navigate through the terminology to browse and discover the full scope of what is there.

Taxonomies for navigation are but one purpose for them to be used in search. Depending on the application domain, richness of the subject matter, scope and depth of topics, these lists can become quite large and complex. The more cross-references (e.g. cell phones USE wireless phones) are embedded in the list, the more likely the searcher's preferred term will be present. There is a diminishing return, however; if the user has to navigate to a system's preferred term too often; the entire process of searching becomes unwieldy and abandoned. On the other hand, if the system automates the smooth transition from one term to another, the richness and complexity of a taxonomy can be an asset.

In more sophisticated applications of taxonomies, the thesaurus model of relationships becomes a necessity. When a search engine, has embedded algorithms that can interpret explicit term relationships, it indexes content according to a taxonomy and all its cross-references. Taxonomy here informs the index engine. It requires substantial maintenance and governance of a much more granular nature than for navigation. To work well, a large corpus of terminology needs to be built to assure that what the content says and means, and what the searcher expects are a match in results. If the results of a search give back unsatisfactory results due to a poor taxonomy, trust in the search system fails rapidly and the benefits of whatever effort was put into building a taxonomy are lost.

I bring this up because the intent of any taxonomy is the first step in deciding whether to start building one. Either model is an on-going commitment but the latter is a much larger investment in sophisticated human resources. The conditions that must be met to have any taxonomy succeed must be articulated in selling the project and value proposition.

Thinking about how product selections are made, prompts me to comment on a two major search product attributes that control the success or failure of search for an enterprise. One is the actual algorithms that control content indexing, what is indexed and how it is retrieved from the index (or indices). The second is the interfaces, interfaces for the population of searchers to execute selections, and interfaces for results presentation. On each aspect, buyers need to know what they can control and how best to execute it for success.

Indexing and retrieval technology is embedded with search products; the number of administrative options to alter search scalability, indexing and content selection during retrieval is limited to none. The "secret sauce" for each product is largely hidden, although it may have patented aspects available for researching. Until an administrator of a system gets deeply into tuning, and experimenting with significant corpuses of content, it is difficult to assess the net effect of delivered tuning options. The time to make informed evaluations about how well a given product will retrieve your content when searched by your select audience is before a purchase is made. You can't control the underlying technology but you can perform a proof of concept (PoC). This requires:

• human resources and a commitment of computing resources


• well-defined amount, type and nature (metadata plus full-text or full-text unstructured-only) to give a testable sample


• testers who are representative of all potential searchers


• a comparison of the results with three to four systems to reveal how well they each retrieve the intended content targets


• knowledge of the content by testers and similarity of searches to what will be routinely sought by enterprise employees or customers


• search logs of previously deployed search systems, if they exist. Searches that routinely failed in the past should be used to test newer systems

In either case, human resource costs must be added to the bottom line. A large number of mature software companies and start-ups are innovating with both their indexing techniques and interface design technologies. They are benefiting from several decades of search evolution for search experts, and now a decade of search experiences in the general population. Search product evolution is accelerating as knowledge of searcher experiences is leveraged by developers. You may not be able to control emerging and potentially disruptive technologies, but you can still exercise beneficial controls when selecting and implementing most any search system.