While speaking on technology portfolio management at the Wilshire/DAMA conference this week, questions about what's behind the famed "Magic Quadrant" came up. Now I certainly don't want to bash Gartner or other analyst firms outright. What they've done to keep vendors somewhat veridical and help IT organizations think through issues is admirable. It's just that their speculative, subjective method is seen increasingly as sub-optimal, particularly when pitted the collective experience of actual in-the-trenches IT professionals.

It's no wonder why IT analysts can achieve rock-star status within the industry, even while playing the same tune (i.e. presenting the same old research) year after year. We're a culture and species that revels in royalty and stardom. Recently researchers at Duke University found that primates will give up treats (or "pay") to see pictures of dominant troop leaders. According to Satoshi Kanazawa, an evolutionary psychologist at London School of Economics, our weakness for endorsements traces back to our "caveman" brains. For better or worse we are hardwired to gush at celebrities (even if Natalie Portman did shave her head for her next role).

But back to quadrants-n-such.

Before you as an IT organization rely too heavily on analyst-generated vendor handicapping methods for selecting IT solutions, or before you as an IT vendor get too jazzed/wigged about where your dot is, ask yourself (or better yet, ask your IT analyst) the following questions:

1. When was the last time the research analyst used or even saw the product operating in the field? How much are they relying on input (e.g. surveys, marketing materials, demos) from the vendors themselves? What quantified user feedback is included?

2. How many dimensions are the vendors rated on? Two? Ten? Is this level of "analysis" really sufficient to discriminate vendors and solutions?

3. Which vendors are absent and why? Many upstart vendors who can't yet ante-up to get on an analyst firm's radar have legitimate offerings.

4. How many total IT solutions are rated in a year? The answer is likely fewer than you have in your entire IT shop.

5. How often are the charts updated? Annually? Bi-annually? Don't vendors come out with new releases (at least minor ones) and new features more often than that?

6. What do real IT pros who really use these products really think? How do they rate their performance, scalability, usability, value, etc? How do they rate the vendor's support, credibility, licensing practices, training, etc?

If you'd like answers to these questions or if you don't know who once said, "The magic quadrant is the most overused, misleading, worst representation of anything," write me at doug.laney@evalubase.com to find out. You'll be quite surprised. Or not.

Just telling IT like it is,
Doug