When a product is described as “the only”, “the best”, the “most complete”, “the fastest,” “the leading,” etc., does anyone actually read and believe these qualifiers in marketing copy, software or otherwise? Do we believe it when an analyst firm writes reviews, or when product hype appears in industry publications?

Most technology buyers have a level of cynicism about such claims and commentary because we know that in each case there is a bias, with good reasons, for the praise. However, also for good reasons, language containing positive sentiment can have an effect – otherwise, it would not be so widespread. At the very least, sentiment analysis tools that are integrated with search engines will pick up on pervasive tones of praise, and from that create new content streams that compound the positive spin.

Being aware of marketing methods and influences on our psyche should arm us with caution but not to a point of being risk averse or frozen to indecisiveness. Instead, we need to find a way to prove the hype and claims through thoughtful, artful and strategic analytical processes. We need methods for testing claims that are appropriate for the solution sought.

First, we need to establish what is appropriate for our business need. Cost is often the primary qualifying factor when narrowing products that will be considered, but this may be short sighted. Business impact and benefits from applying the right solution need to be directly in our line of sight. If the solution you acquire can be evaluated to demonstrate a significant business benefit, the cost of a higher priced product may also be high-value to your business. Add to business impact the scope for the use of an enterprise search engine (how widely deployed and leveraged) and whether it can scale to include multiple searchable repositories across the organization; these attributes may enhance business impact.

Judging business impact, scope and scaling enterprise search products is a tricky proposition. You absolutely cannot do it by totaling the number of positive checks a vendor ticks off on a spreadsheet of requirements. While such a device can be useful for narrowing down a field of products to those you might select, it is only a beginning. All too often, this is where the selection process ends.

What needs to be done next? I recommend these evaluation steps that can be done concurrently:

  • Find customers by using social tools; reading and researching. With so many Web-based social and search tools it should be easy to identify individuals and enterprises that are actually users of products you are considering. Reach out, schedule talk time and have a pointed list of questions ready to investigate their experiences – listen carefully and follow up on any comments that sound a note of caution.
  • Run a proof-of-concept initiative that includes serious testing by key users with content that is relevant to the testers. Develop test cases and define explicitly for the testers what they are searching, and what you want to learn.
  • Keep careful notes throughout your interactions with vendors, as you seek information, test their products and request answers to technical questions. The same goes for the conversations with their customers, the ones you find on your own, not just the ones vendors steer you to. Your inquiry needs to include information about business relationship issues, responsiveness, ease of use, and how well a vendor can understand and respond to your business needs in these early relationship stages.
  • If things are not going smoothly, observe how a vendor reacts and responds; what is their follow-up and follow-through in the pre-purchase stage? Never succumb to the excuse that because they are “going through growing pains,” have “so much business demand” they are stretched thin, or that something else is more important to them than your product evaluation. If any of these creep in before you purchase, you have a major symptom conveying clearly that your business is not as important to the vendor or not as valuable as another company’s.

Longevity of use of an enterprise search application must be foremost in your mind throughout all of these steps. While many enterprises try to plan for upgrading or replacing legacy software applications to remain competitive and current using newer technologies, actual experiences are rarely ideal. You could be “stuck” with your choice for a decade or longer. Being in a dependent relationship with a vendor or product you are not happy with, will be a miserable experience and no benefit to your enterprise, no matter how popular the product is in the industry press.

The steps for selection will take a little longer than just sending out RFPs and reading responses, but it is really worth it over the long haul relationship you are about to engage.