The topic of business data quality has been with us for decades. Given the large number of vendor offerings dedicated to resolving data quality (DQ) issues, one might be forgiven for believing that the problems have all but been resolved. A glance through the current literature reveals, however, that the problem of poor data quality is still very much alive.
Has the state of data quality in organizations improved over the past two decades? Or is data management, and data quality in particular, still in a ghastly state with organizations and senior management feeling that the problem is overwhelming? Despite the clear concerns from business and a plethora of software vendors, there is surprisingly little concrete information available regarding the state of data quality in business.
At The Information Difference we believe it is important for both organizations and vendors to understand the current state of data quality in organizations. We have therefore conducted a survey, sponsored by Pitney Bowes Business Insight and Silver Creek Systems, aimed at gaining deeper insight into the views of businesses regarding their current or planned data quality initiatives.
Some 193 respondents completed the survey from all around the world, the majority from Europe (47%) and North America (44%). A high proportion (39%) of the respondents were from companies having annual revenues greater than US $ 1 billion; respondents represented a wide spectrum of industries.
Some of the main findings from the survey are summarized below:
One third of respondents rate their data quality as poor at best and only 4% as excellent. Fully half considered their data quality as good, although this may be somewhat over-optimistic when set against other results from the survey. For example one respondent told us “Poor data quality and consistency has led to the orphaning of $32 million in stock just sitting in the warehouse that can’t be sold since it’s lost in the system.”
63% have no idea what poor data quality may be costing them.
Surprisingly, 17% have no plans at all to start a data quality initiative, compared with 37% who currently have some form of data quality initiative in place. The remainder plan to introduce data quality in the next one to three year period.
Some two-thirds plan for, or currently have, data quality spanning either the entire enterprise or one or more lines of business.
A remarkable 81% say that their data quality is focused wider than just “name and address” yet this latter is the area in which most (>90%) vendors currently have their base!
The top three data areas for DQ were ranked as: 1) product data; 2) financial data; 3) name and address data. It is interesting that financial data occupies second place but virtually no current DQ vendors specialize in this area. Product data is rated as a higher priority than customer name and address, yet only a few data quality vendors specialise in dealing with product data quality.
The top two barriers to adopting data quality were: Management does not see this as an imperative and It’s very difficult to present a business case. This is interesting given that the majority (63%) have not attempted to calculate the cost of data errors.
We automatically stop accepting comments 180 days after a post is published. If you would like to know more about this subject, please contact us and we'll try to help.
30th July 2009: 'Tommy Drummond, Informatica' said:
Mmmmm. “Business data quality”? Has it really been with us for decades? I don’t think so! However I do believe we are now entering a new phase where “business data quality” can exist and will deliver the benefits so badly needed by the business. Let me explain. First, the business needs to acknowledge a specific set of responsibilities e.g. defining data quality targets, documenting data quality business rules, managing exceptions and monitoring data quality. The business needs to assign roles and responsibilities for data quality per business unit and per process. As an example, let us consider a global organization with a data quality steward per country assigned responsibility for ensuring all processes execute error free based on high quality data. So what does this person do?
On a Monday morning, the data quality steward receives an email with a link to a data quality scorecard for all key processes and applications within the business unit. The data quality steward follows the link to an intranet site which highlights the trends and also highlights all exceptions, categorized by data quality dimension (completeness, conformity, consistency, duplicate, etc.), data type (customer data, product, financial, asset) and their impact on the business. The next task for the data quality steward is to remedy the situation, which involves profiling data, editing reference data, reconfiguring a rule and testing the rule. He/she will also need to collaborate with IT to address some of the data quality rules and the testing cycle. This can be done within a matter of days.
The next Monday morning, the data quality steward looks forward to his/her email and data quality scorecard – which shows an improvement based on the newly configured rules and updated reference data.
So is this the standard “business data quality” process in most organizations? I don’t believe so. But I can see that the prerequisites are being put in place for this scenario to become the standard. These prerequisites include (i) easy to use business UIs – preferably browser based, to support all responsibilities of a data quality steward (ii) collaborative tools to simplify and speed up specification cycles with IT and (iii) a unified data quality and data integration platform to ensure that once the data quality rules are built, they can be deployed across the organization to all relevant applications and processes.
And will business data quality deliver the goods? Yes, the business case becomes compelling when the daily exceptions can be linked to business value – and that in my opinion is what business data quality is all about.
Reply to Tommy Drummond, Informatica?