Recently in Search Technologies and Products Category
This topic is random and a short response to a question that popped up recently from a reader seeking technical research on the subject. Since none was available in the Gilbane library of studies, I decided to think about how to answer the subject with some practical suggestions.
The focus is on an enterprise with a substantive amount of content aggregated from a diverse universe of industry specific information, and what to do about searching it. If the information has been parsed and stored in an RDBMS database, is it not better to leverage the SQL query engine native to the RDBMS? Typical database engines might be: DB2, MS Access, MS SQL, MySQL, Oracle or Progress Software.
To be clear, I am not a developer but worked closely with software engineers for 20 years when I owned a software company. We worked with several DBMS products, three of them supported SQL queries and the application we invented and supported was a forerunner of today's content management systems with a variety of retrieval (search) interfaces. The retrievable content our product supported was limited to metadata plus abstracts up to two or three pages in length; the typical database sizes of our customers ranged from 250,000 to a couple of million records.
This is small potatoes compared to what search engines typically traverse and index today but scale was always an issue and we were well aware of the limitations of the SQL engines to support contextual searching, phrase searching and complex Boolean queries. It was essential that indexes be built in real time, when records were added whether manually through screen forms, or through batch loads. The engine needed to support explicit adjacency (phrase) searching as well as key words anywhere in a field, in a record, or in a set. Saving and re-purposing results, storing search strategies, narrowing large sets incrementally, and browsing indexes of terminology (taxonomy navigation) to select unique terms that would enable a Boolean "and" or "or" query were part of the application. When our original text-based DBMS vendor went belly-up, we spent a couple of years test driving numerous RDBMS products to find one that would support the types of searches our customers expected. We settled on Progress Software primarily because of its support for search and experience as an OEM to application software vendors, like us. Development time was minimized because of good application building tools and index building utilities.
So, what does that have to do with the original question, native RDBMS search vs. standalone enterprise search? Based on discussions and observations with developers trying to optimize search for special applications, using generic search tools for database retrieval, I would make the following observations. Search is very hard and advanced search, including concept searching, Boolean operations, and text analytics, is harder still. Developers of enterprise search solutions have grappled with and solved search problems that need to be supported in environments where content is dynamically changing and growing, different user interfaces for diverse audiences and types of queries are needed, and query results require varieties of display formats. Also, in e-commerce applications, interfaces require routine screen face lifts that are best supported by specialized tools for that purpose.
Then you need to consider all these development requirements; they do not come out-of-the-box with SQL search:
- Full text indexes and database field or metadata indexes require independent development efforts for each database application that needs to be queried.
- Security databases must be developed to match each application where individual access to specific database elements (records or rows) is required.
- Natural language queries require integration with taxonomies, thesauri, or ontologies; this means software development independent of the native search tools.
- Interfaces must be developed for search engine administrators to make routine updates to taxonomies and thesauri, retrieval and results ranking algorithms, adjustments to include/exclude target content in the databases. These content management tasks require substantive content knowledge but should not require programming expertise and must be very efficient to execute.
- Social features that support interaction among users and personalization options must be built.
- Connectors need to be built to federate search across other content repositories that are non-native and may even be outside the enterprise.
Any one of these efforts is a multi-person and perpetual activity. The sheer scale of the development tasks mitigate against trying to sustain state-of-the-art search in-house with the relatively minimalist tools provided in most RDBMS suites. The job is never done and in-depth search expertise is hard to come by. Software companies that specialize in search for enterprises are also diverse in what they offer and the vertical markets they support well. Bottom line: identify your business needs and find the search vendor that matches your problem with a solution they will continue to support with regular updates and services. Finally, the issue of search performance and speed of processing are another huge factor to consider. For this you need some serious technical assessment. If the target application is going to be a big revenue generator with heavy loads and huge processing, do not overlook. Do benchmarks to prove the performance and scalability.
It would be presumptuous to think that I could adequately summarize a very active year of evolution among a huge inventory of search technologies. This entry is more about what I have learned and what I opine about the state-of-the-market, than an analytical study and forecast.
The weak link in the search market is product selection methods. My first thought is that we are in a state of technological riches without clear guideposts for which search models work best in any given enterprise. Those tasked to select and purchase products are not well-educated about the marketplace but are usually not given budget or latitude to purchase expert analysis when it is available. It is a sad commentary to view how organizations grant travel budgets to attend conferences where only limited information can be gathered about products but will not spend a few hundred dollars on in-depth comparative expert analyses of a large array of products.
My sources for this observation are numerous, confirmed by speakers in our Gilbane conference search track sessions in Boston and San Francisco. As they related their personal case histories for selecting products, speakers shared no tales of actually doing literature searches or in-depth research using resources with a cost associated. This underscores another observation, those procuring search do not know how to search and operate in the belief that they can find "good enough" information using only "free stuff." Even their review of material gathered is limited to skimming rather than a systematic reading for concrete facts. This does not make for well-reasoned selections. As noted in an earlier entry, a widely published chart stating that product X is a leader does nothing to enlighten your enterprise's search for search. In one case, product leadership is determined primarily by the total software sales for the "leader" of which search is a miniscule portion.
Don't expect satisfaction with search products to rise until buyers develop smarter methods for selection and better criteria for making a buy decision that suits a particular business need.
Random Thoughts. It will be a very long time before we see a universally useful, generic search function embedded in Microsoft (MS) product suites as a result of the FAST acquisition. Asked earlier in the year by a major news organization whether I though MS had paid too much for FAST, I responded "no" if what they wanted was market recognition but "yes" if they thought they were getting state-of-the-art-technology. My position holds; the financial and legal mess in Norway only complicates the road to meshing search technology from FAST with Microsoft customer needs.
I've wondered what has happened to the OmniFind suite of search offerings from IBM. One source tells me it makes IBM money because none of the various search products in the line-up are standalone, nor do they provide an easy transition path from one level of product to another for upward scaling and enhancements. IBM can embed any search product with any bundled platform of other options and charge for lots of services to bring it on-line with heavy customization.
Three platform vendors seem to be penetrating the market slowly but steadily by offering more cohesive solutions to retrieval. Native search solutions are bundled with complete content capture, publishing and search suites, purposed for various vertical and horizontal applications. These are Oracle, EMC, and OpenText. None of these are out-of-the-box offerings and their approach tends to appeal to larger organizations with staff for administration. At least they recognize the scope and scale of enterprise content and search demands, and customer needs.
On User Presentations at the Boston Gilbane Conference, I was very pleased with all sessions, the work and thought the speakers put into their talks. There were some noteworthy comments in those on Semantic Search and Text Technologies, Open Source and Search Appliances.
On the topic of semantic (contextual query and retrieval) search, text mining and analytics, the speakers covered the range of complexities in text retrieval, leaving the audience with a better understanding of how diverse this domain has become. Different software application solutions need to be employed based on point business problems to be solved. This will not change, and enterprises will need to discriminate about which aspects of their businesses need some form of semantically enabled retrieval and then match expectations to offerings. Large organizations will procure a number of solutions, all worthy and useful. Jeff Catlin of Lexalytics gave a clear set of definitions within this discipline, industry analyst Curt Monash provoked us with where to set expectations for various applications, and Win Carus of Information Extraction Systems illustrated the tasks extraction tools can perform to find meaning in a heap of content. The story has yet to be written on how semantic search is and will impact our use of information within organizations.
Leslie Owens of Forrester and Sid Probstein of Attivio helped to ground the discussion of when and why open source software is appropriate. The major take-way for me was an understanding of the type of organization that benefits most as a contributor and user of open source software. Simply put, you need to be heavily vested and engaged on the technical side to get out of open source what you need, to mold it to your purpose. If you do not have the developers to tackle coding, or the desire to share in a community of development, your enterprise's expectations will not be met and disappointment is sure to follow.
Finally, several lively discussions about search appliance adoption and application (Google Search Appliance and Thunderstone) strengthen my case for doing homework and making expenditures on careful evaluations before jumping into procurement. While all the speakers seem to be making positive headway with their selected solutions, the path to success has involved more diversions and changes of course than necessary for some because the vetting and selecting process was too "quick and dirty" or dependent on too few information sources. This was revealed: true plug and play is an appliance myth.
What will 2009 bring? I'm looking forward to seeing more applications of products that interest me from companies that have impressed me with thoughtful and realistic approaches to their customers and target audiences. Here is an uncommon clustering of search products.
Multi-repository search across database applications, content collaboration stores document management systems and file shares: Coveo, Autonomy, Dieselpoint, dtSearch, Endeca, Exalead, Funnelback, Intellisearch, ISYS, Oracle, Polyspot, Recommind, Thunderstone, Vivisimo, and X1. In this list is something for every type of enterprise and budget.
Business and analytics focused software with intelligence gathering search: Attensity, Attivio, Basis Technology, ChartSearch, Lexalytics, SAS, and Temis.
Comprehensive solutions for capture, storage, metadata management and search for high quality management of content for targeted audiences: Access Innovations, Cuadra Associates, Inmagic, InQuira, Knova, Nstein, OpenText, ZyLAB.
Search engines with advanced semantic processing or natural language processing for high quality, contextually relevant retrieval when quantity of content makes human metadata indexing prohibitive: Cognition Technologies, Connotate, Expert System, Linguamatics, Semantra, and Sinequa
Content Classifier, thesaurus management, metadata server products have interplay with other search engines and a few have impressed me with their vision and thoughtful approach to the technologies: MarkLogic, MultiTes, Nstein, Schemalogic, Seaglex, and Siderean.
Search with a principal focus on SharePoint repositories: BA-Insight, Interse, Kroll Ontrack, and SurfRay.
Finally, some unique search applications are making serious inroads. These include Documill for visual and image, Eyealike for image and people, Krugle for source code, and Paglo for IT infrastructure search.
This is the list of companies that interest me because I think they are on track to provide good value and technology, many still small but with promise. As always, the proof will be in how they grow and how well they treat their customers.
That's it for a wrap on Year 2 of the Enterprise Search Practice at the Gilbane Group. Check out our search studies at http://gilbane.com/Research-Reports.html and PLEASE let me hear your thoughts on my thoughts or any other search related topic via the contact information at http://gilbane.com/contact.html
Meaning is a very large concept in every aspect of search technology and dozens of search product sites include either the words "semantic" or "meaning" as a key element of the offered technology. This is not as far fetched as search product claims to "know" what the searcher wants to find, as if "knowing" can be attributable to non-human operations. However, how well a search engine indexes and retrieves content to meet a searcher's intent, is truly in the eyes of the beholder. I can usually understand why, technically speaking, a piece of content turns up in a search result, but that does not mean that it was a valid scrap for my intent. My intent for a search cannot possibly be discernible by a search engine if, as is most often the case, I don't explicitly and eloquently express what, why, and other contextual facts when entering a query.
The session we have set aside at Gilbane San Francisco for a discussion on current activity related to semantic technologies will undoubtedly reveal more meaning about technologies and art of leveraging tools to elicit semantically relevant content. I suspect that someone will also stipulate that what works requires a defined need and clear intent during the implementation process - but what about all those fuzzy situations? I hope to find out.
This is the last posting before the conference this week so I hope you will add this enterprise search session (EST-6: Semantic Technology - Breakdown or Breakthrough) being moderated by Colin Britton to your agenda on June 19th. He will be joined by speakers: Steve Carton, VP Content Technologies, Retrieval Systems Corp., Folksonomies: Just Good Enough for all Kinds of Thing, Prakesh Govindarajulu, President, RealTech Inc, Building Enterprise Taxonomies from the Ground Up, and Jack Jia, Founder & CEO, Baynote.
See you in San Francisco in person or virtually thereafter.
Executives from four companies with unique products for solving search challenges within organizations will share their thoughts on what is most interesting, promising and problematic in the current market at Gilbane San Francisco, session EST-4 on June 19th. Because it is important to consider markets, this type of session gives the panel and audience a chance to recognize the perspectives of each other, sellers and buyers. I'll be asking the executives to comment on their product strengths, with emphasis on specific value propositions for buyers. There will be an opportunity for the audience to ask questions because what is on the minds of buyers is often the most interesting for everyone to hear. When one listens to questions asked of vendors and hears responses, it reveals two important things about markets:
- Alignment that exists (or not) between what buyers want and what sellers have to offer
- Challenges both face to create value
No one, particularly vendors, likes to talk about negatives. However, given all the things that can go wrong for a buyer trying to manage an implementation team or for sellers who don't anticipate expectations about their products not explored during the selection process, it is important to consider factors that lead to failure or less than satisfactory outcomes. One question I will have for vendors is to share honestly some of the challenges or disappointments about the market that are a particular point of pain for them. Bad-mouthing customers is not the answer but conveying how hard most vendors honestly try to create value and how their best intentions can be derailed through miscommunication may help buyers and sellers smooth the communication flow.
After all, don't we all want to provide value for our internal and external customers? I think you will find the panel a receptive group: a 25+-year veteran of the information access market, a marketing executive with an international search and text analytics firm, a founder of a rapidly growing plug-and-play search solution, and a marketing VP working to position a company with a large scale solution against the "big-three" search solutions. You'll hear straight talk and interesting value propositions.
Speakers: Margie Hlava, President, Accession Innovation, David Haucke, VP Global, Marketing, ISYS, Laurent Simoneau, President & CEO, and Rebecca Thompson, VP Marketing, Vivisimo
Since I weighed in last week on the Microsoft acquisition of FAST Search & Transfer, I have probably read 50+ blog entries and articles on the event. I have also talked to other analysts, received emails from numerous search vendors summarizing their thoughts and expectations about the enterprise search market and had a fair number of phone calls asking questions about what it means. The questions range from “Did Microsoft pay too much?, to “Please define enterprise search,” to “What are the next acquisitions in this market going to be going to be?” My short and flippant answers would be “No,” “Do you have a few hours?” and “Everyone and no one.”
I have seen some excellent analysis contributing relevant commentary to this discussion, some misinterpretation of what the distinction’s are between enterprise search and Web search, and some conclusions that I would seriously debate. You’ll forgive me if I don’t include links to the pieces that influenced the following comments. But one by Curt Monash in his piece on January 14 summarized the state of this industry and its already long history. It is noteworthy that while the popular technology press has only recently begun to write about enterprise search, it has been around for decades in different forms and in a short piece he manages to capture the highlights and current state.
Other commentary seems to imply that Microsoft is not really positioning itself to compete with Google because Google is really about Web (Internet) searching and Microsoft is not. This implies that FAST has no understanding of Web searching. Several points must be made:
1. FAST Search & Transfer has been involved in many aspects of search technologies for a decade. Soon after landing on our shores it was the search engine of choice for the U.S. government’s unifying search engine to support Internet-based searching of agency Web sites by the public. Since then it has helped countless enterprises (e.g. governments, manufacturers, e-commerce companies) expose their content, products and services via the Web. FAST knows a lot about how to make Web search better for all kinds of applications and they will bring that expertise to Microsoft.
2. Google is exploiting the Web to deliver free business software tools that directly challenge Microsoft stronghold ( e.g. email, word processing). This will not go unanswered by the largest supplier of office automation software.
3. Google has several thousand Google Enterprise Search Appliances installed in all types of enterprises around the world, so it is already as widely deployed in enterprises in terms of numbers as FAST, albeit at much lower prices and for simpler application. That doesn’t mean that they are not satisfying a very practical need for a lot of organizations where it is “good enough.”
For more on the competition between the two check this article out.
Enterprise search has been implied to mean only search across all content for an entire enterprise. This raises another fundamental problem of perception. Basically, there are few to no instances of a single enterprise search engine being the only search solution for any major enterprise. Even when an organization “standardizes” on one product for its enterprise search, there will be dozens of other instances of search deployed for groups, divisions, and embedded within applications. Just two examples are the use of Vivisimo now used for USA.gov to give the public access to government agency public content, even as each agency uses different search engines for internal use. Also, there is IBM, which offers the OmniFind suite of enterprise search products, but uses Endeca internally for its Global Services Business enterprise.
Finally, on the issue of expectations, most of the vendors I have heard from are excited that the Microsoft announcement confirms the existence of an enterprise search market. They know that revenues for enterprise search, compared to Web search, have been miniscule. But now that Microsoft is investing heavily in it, they hope that top management across all industries will see it as a software solution to procure. Many analysts are expecting other major acquisitions, perhaps soon. Frequently mentioned buyers are Oracle and IBM but both have already made major acquisitions of search and content products, and both already offer enterprise search solutions. It is going to be quite some time before Microsoft sorts out all the pieces of FAST IP and decides how to package them. Other market acquisitions will surely come. The question is whether the next to be acquired will be large search companies with complex and expensive offerings bought by major software corporations. Or will search products targeting specific enterprise search markets be a better buy to make an immediate impact for companies seeking broader presence in enterprise search as a complementary offering to other tools. There are a lot of enterprise search problems to be solved and a lot of players to divvy up the evolving business for a while to come.
I closed 2007 with some final takeaways from the Gilbane Conference and notes about semantic search. Already we are planning for Gilbane San Francisco and you are invited to participate. There is no question that enterprise search, in all its dimensions, will be a central theme of several sessions at the conference, June 17th through 19th. I will lead with a discussion in which a whole range of search topics, technologies and industry themes will be explored in a session featuring guest Steve Arnold, author of Google Version 2.0, The Calculating Predator. To complement the sessions, numerous search technology vendors will be present in the exhibit hall.
A most important conference component will be a highlight for conference goers, shared-experiences about selecting, implementing and engaging with search tools in the enterprise. Everyone wants to know what everyone else is doing, learning and what they know about enterprise search. You may want to present your experiences or those of your organization. If you are interested, considering presenting, know of a good case study, usability or “lessons learned” from implementing search technology, please raise your hand. You can do this by reaching out through this link to submit a proposal and make reference to the “enterprise search blog call for papers.” You can be sure I’ll follow-up soon to explore the options for you or a colleague to participate. This is a great opportunity to be part of a community of practitioners like you and attend a conference that always has substantive value for participants.
Leave it to Microsoft to end the year with a big announcement and open the next one with an even bigger one. We knew that the world of enterprise search was going to contract in terms of the number of established vendors, even though it is expanding in new and innovative offerings. Microsoft had to make a bold play in an industry where Google has been the biggest player on the WWW stage while reaching deeper into the enterprise, tickling at Microsoft’s decades-old hold on content creation and capture. So, with the acquisition of FAST Search & Transfer, whose technology may not be the best in the enterprise search market but is certainly the most widely deployed at the high-end, Microsoft opens with a direct challenge to its largest competitor.
Boy! Have the emails been flying this morning. At least I know there will be plenty of material to ponder in the next few weeks and months. P.S. Don’t miss the action in San Francisco!
In a week when the KMWorld and Enterprise Search Summit were running concurrently in San Jose, Microsoft made an enterprise search product announcement that was actually a well-kept secret for Microsoft. There was plenty of other new product news floating about the marketplace, too. Mark Logic, MuseGlobal, Cognos, SchemaLogic, and Brainware all had their own announcements.
Between November 6 and November 13, these five companies had interesting news to share. The announcements all related to leveraging enterprise content in tandem with search engines. This underscores a strong trend in software product deployment, specifically, that much of it is being rolled out in partnerships in highly heterogeneous environments. While Microsoft’s announcement about free Search Server 2008 Express establishes them as the last major software company to adopt search as a platform, the other technology announcements remind us that integration activity is a core operational consideration and even a necessity for gaining value from search.
In order to tie all the bits and pieces of content across the enterprise into a tidy bundle for simple retrieval, or in order for content to really bring value to solving business problems, it needs packaging. It needs to be packaged at the front end so that search engines can grab useful context and metadata for smarter indexing. It also needs to be well packaged at the output end to present results meaningfully for a particular audience or purpose.
Here is a quick look at what these five complementary technologies do for search plus a link to each of their latest announcements:
> Brainware – combines data capture with a content extraction and distillation learning engine for enhancing categorization relevancy in preparation for natural language queries. It will be embedded in search for a leading enterprise library system, Sirsi/Dynix.
> Cognos – a leading Business Intelligence (BI) software company is being acquired by IBM, whose search products are often paired with Cognos.
> Mark Logic – is a company with an XML content server platform for managing or converting content in XML formats. They just announced MarkMail, a community-focused searchable message archive service, which stores emails as XML documents. Expect more from them on this front.
> MuseGlobal – offers solutions that integrate content from multiple search engines. They just announced availability for presenting results in a fully unified and consistent format from multiple search engines in a SharePoint portal interface.
> SchemaLogic – specializes in content and document type modeling, metadata and vocabulary management using SchemaServer. In the past two weeks they have announced integration with SharePoint to manage metadata. A webinar this week described the interplay with Documentum for document production and retrieval using the FAST search engine.
And what do the other enterprise search vendors have to say about the “surprise” Microsoft announcement? Comments ranged from “we knew it was just a matter of time before they announced” to “good for business, enterprise search is officially now a market.” To the first comment I say, “Not so fast.” For several years rumors have been floated about the imminent acquisition of any number of search companies by MS but nothing materialized. Yes, Microsoft was doing something about enterprise search but until last week “what” was still the question. To the latter I say, “We’ve had an enterprise search market for several years, Microsoft just wanted to be sure it was well established before joining the club.” That was smart of them; let others lay the foundation for a growth industry. It also looks like this is a leveling of the field with Google already playing in Microsoft’s backyard in the free office tools area.
Now the positioning really begins.
I am in a mode of indecision about prioritizing a lot of news in the enterprise search space; it all seems important because we have an agenda focused on search at the upcoming conference on November 27 – 29 in Boston. The following, in no particular order, is not an exhaustive list but representative of happenings in the past month that will surely be the subject of much commentary and discussion by our speakers and panelists:
> Don Dodge of Microsoft, one of our panelists in the search keynote session, is taking on Google’s customer support positioning as an enterprise solutions provider in his blog.
> MondoSoft went from being shut down by its investors to being acquired by possibly two companies (one acquiring MondoSearch and the other acquiring Ontolica) and now both are being acquired by one company. Meanwhile, IntelliSearch is making an offer to MondoSearch clients to “switch and save.”
> Steve Arnold is continuing his drumbeat on Google search patents and their significance.
> Oracle is getting very serious about search as you will hear in this Webinar download and is positioning itself for a holistic approach to managing content in the enterprise.
> Fast and Autonomy are making acquisitions, too, and have begun to act like they are not the only search options for the enterprise by re-focusing their marketing.
> Companies like Connotate, ISYS, Exalead, Recommind, SchemaLogic and Coveo are acquiring good clients and showing their strengths in important niche markets with new enhancements
> Vivisimo’s social search is getting a lot of positive press, inviting a lot of blogging and has set the bar high for competitors to match their offerings.
> Endeca continues to expand its staff, client-base and well-engineered product line; they are also building important alliances with technology and business partners when it makes good sense to do so
> IBM is talking up the potential of semantic search in the enterprise.
Whew!
Most of the above mentioned will be making an appearance or two at GilbaneBoston, as speakers or exhibitors, or both. I am trying to figure out how to make sure the seven sessions on search and semantic Web technologies touch most of the bases but with so much afoot in the search arena, we will be working overtime. If you are going to be an attendee in any capacity, I hope you’ll blog or make comments when I do. We want to hear what you think and learn about from the experts and users alike. There are sure to be surprises. Your take on the programs will be of interest to many. If you do make the conference, be sure to find me and introduce yourself so we can have a chat.
Drew Robb has written an excellent article that reflects new thinking about the convergence of older and emerging technologies. In Search Converging with Business Intelligence for CRM Daily.com on August 28, 2007 I see similarities to my own way of viewing what is possible with newer search tools. Be sure to read it.
In addition to enterprise search, I actively follow enterprise knowledge management. It has been much debated because of confusion about its links to inappropriate technologies and well-intentioned but costly and failed initiatives in many organizations. But, in spite of rumors about its death, KM will remain a boundless frontier of opportunity. At its best, it leverages collaborative and sharing practices to maximize the value of organizations’ discoveries, developments and learning using innovative and often simple practices that work because they suit a particular culture’s way of operating.
Popular writings about business and technology innovation, plus tools and techniques for collaboration and sharing abound. 2007 is surely the year when search has come to dominate the technology landscape as vendors in BI, text mining and text analytics, data management, and countless semantic and Web 2.0 entrants vie to add refinements, and conversely search integrates features from those technologies.
In August I commented on search offerings that have made a point of highlighting their “Sharepoint” connectivity. Similarly, many products are adding claims for exploiting emails. I have long assumed that email should be part of the search engine crawling and indexing mix for any intranet. Given email structure, it seems to have more useful and usable metadata than a lot of other content. Social network analysis tools have been terrific at revealing fascinating relationships and internal communications within organizations, especially in the discovery area as emails have been a source for exploring questionable business practices in legal proceedings.
More sophisticated analytic and semantic techniques for exploiting concepts in content give hints about how technologies can integrate content by mapping experts to the expertise they contribute, even when it is scattered throughout their work, including emails where so many nuggets may reside. An area for development would correlate nuggets of knowledge in emails to reveal hidden and latent expertise, pointing to other content an individual has produced using search with BI and analytics algorithms.
Maybe I’m overreaching but I suspect that a lot of experts may not be sufficiently motivated, disciplined or expected to aggregate their small but useful contributions into more valuable knowledge. Regardless of the reasons for that failure, much could be revealed with the right blending of search, indexing, analytics and business intelligence technologies. The components already exist but implementation to get desired results are not necessarily easy to deploy. A truly innovative expertise exploitation engine would be a knowledge engine of note, able to synthesize new knowledge in unique and interesting ways. Historically, much has been made about the role of serendipity in the “search for truth” and “quest for knowledge.” With the aid of enhanced search technologies to blend any or many expert nuggets, a lot more serendipity might happen.
As we gear up for Gilbane Boston 2007, the number of possible topics to include in the tracks related to search seems boundless. The search business is in a transitional state but in spite of disarray is still pivotal in its impact on business and current culture. The sessions will reflect the diversity in the market.
One trend is quite clear; the amount of money and effort being expended for Web search or site search on commercial Web sites is a winner in the “search technology” revenues war with annual revenues measuring well into the $billions. On the other hand, a recent Gartner study described the 2006 revenues for enterprise search as below $400M. This figure comes from reading an excellent article, Enterprise Search: Seek and Maybe You’ll Find, by Ben DuPont in Intelligent Enterprise. Check it out.
The distinctions between search on the Web and search within the enterprise are numerous but here are two. First, Internet Web search revenue is all about marketing. Yes, we use it to discover, learn, find facts, and become more informed. But when companies supplying search technology to expose you to their content on the Internet they do so to facilitate commerce. If it falls into the hands of organizations that have other intent, libraries or government agencies, so be it.
As we all know, when we are at work, seeking to discover, learn or find facts to do our jobs better, we need a different kind of search. Thus, we seek a clear search winner built just for our enterprise with all of its idiosyncrasies. The problem is that what is inside does not look like the rest of the world’s content as it is aggregated for commercial views. Enterprises are unique and operate sometimes chaotically, or, at best, with nuanced views of what information is most important.
The second distinction relates to taxonomies, and the increase in their development and use. I’ve seen a dramatic increase in job postings for “taxonomists” and have managed several projects for enterprises over the years to build these controlled lists of terms for categorizing content. What is noteworthy about recent job opportunities is that most seem to be for customer facing Web sites. Historically, organizations with substantial internal content (e.g. research reports, patents, laboratory findings, business documents) hired professionals to categorize materials for a narrowly defined audience of specialists. The terminology was often highly unique, could number in the hundreds or thousands of terms, even for a relatively small enterprise. This is no longer a common practice.
Slow financial growth in enterprise search markets is no surprise. Like many tools designed and marketed for departments not directly tied to revenue generation, search goes begging for solid vertical markets. Search’s companion technologies are also struggling to find a lucrative toehold for use within the organization. Content management systems integrated with rich and efficient taxonomy building and maintenance functions are hard to find.
I am confident that tools in CMS products for building and maintaining complex taxonomies will not improve until enterprises find a solid business reason to put professional human resources into doing content management, taxonomy development, search, and text analytics on their most important knowledge assets. This is a tough business proposition compared to the revenues being driven on the Internet. What businesses need to keep in mind is that without the ability to leverage their internal knowledge content assets better, smarter and faster, there won’t be innovative products in the pipeline to generate commerce. Losing track of your valuable intellectual resources is not a good long term strategy. Once you begin committing to solid content resource management strategies, enterprise technology products will improve to meet your needs.
Now available! "Beyond Search: What to do When Your Enterprise Search System Doesn't Work, by Stephen Arnold
