Here at Bloor Research we have recently been investigating why so many data migration projects (84%) run over time or budget. Over half of the respondents in our survey who had run over budget blamed inadequate scoping (that is, they had not properly or fully profiled their data) and more than two thirds of those that had gone over time put their emphasis in the same place.

I mention this because it is symptomatic of all data integration and data movement projects: data quality needs to start before you begin your project (so that you can properly budget time and resources) and continues right through the course of the project and, where it is not a one-time project like data migration, is maintained on an on-going basis through production. Further, in order to maintain quality you need to be able to monitor data quality (via dashboards and the like) on an on-going basis as well. This is especially important within the context of data governance.

In other words, data quality follows the lifecycle of your data, and it spans multiple applications and systems as the data is reused and shared. In this article I discuss the need for an integrated data quality platform to support such an environment, with particular reference to the Trillium Software System, whose latest release (version 11) has just been announced.

Version 11 includes some very significant features, not least of which is that this release represents the culmination of the efforts the company has made, over the last few years, to fully integrate its Avellino acquisition. Specifically, this means that there is now a single repository (Metabase) which is shared across the environment (or you can have multiple Metabases), and that there is a single integrated interface across the product. The consequence of this is that data profiling and analysis need not be distinct from data cleansing and matching. In other words, you can now easily swap between one function and the other, as requirements dictate, rather than being forced to use a more waterfall-style approach in which profiling came first and quality came second.

The second major enhancement is the introduction of phrase analysis. This provides the ability to identify unique words and phrases (substrings), and combinations thereof, within a selected attribute. The importance of this is that it provides the ability to parse unstructured and semi-structured data and to then build data quality rules based on these. This is particularly important if you want to apply data quality rules to product data or other descriptive data that comes into the organisation in unstructured formats.

Shortly, phrase analysis will be extended by the introduction of Universal Data Libraries that will provide standardised taxonomies for things such as colours, units of measure, currencies, sizes and shapes and so on. Although the core libraries will be in English at first, you will be able to customise these libraries so that you can automatically recognise that blau = bleu = blue for example.

In addition, the company has opened up the Metabase so that customers and partners can more easily integrate data quality and profiling processes into their applications and in order to provide a foundation for interactive report building. Through this API, for example, customers can easily incorporate metadata from Trillium, such as profiling results or business rules, into broader metadata repositories or other applications.

Finally, this release sees the introduction of time series analysis. Previously, you could only take snapshots of quality information but, with time series support, trending becomes possible so that you can monitor data quality and profiling statistics over time.

Anyway, so much for the major new features (there are a number of smaller ones as well) in version 11. Now I want to focus more on strategy.

There are two types of data quality vendor: pure plays (like Trillium) and those integrated with ETL tools.

As far as pure plays in the market are concerned, most of these (apart from Trillium) are smaller companies that specialise in a particular area such as real-time data quality (embedding data quality into call centre applications) or product data quality, for example. In contrast, Trillium is recognised as an enterprise-wide solution, providing the tools and content to support data quality needs across a wide range of business applications, data domains, and implementation types. However, there is more to it than that: by choosing the smaller suppliers of point solutionswhether for price or because they offer deeper capabilities in a specific areayou will end up having multiple data quality tools when it might be better to have a single solution that did everything, even if, in some circumstances, it wasnt quite the best thing since sliced bread.

Now, this may seem like the traditional integrated solution versus best-of-breed argument. Some people like one, some people like the other. However, in the case of data quality it is not quite as simple as that because using a data quality platform such as Trilliums allows you to reuse business rules across both real-time and batch processes, and across different application environments. Further, if we consider the implementation of data governance then one of the precepts involved would be the adoption of common data quality standards across the organisation, which is best facilitated by using a common platform and reusable rules.

For example, consider the respective requirements for data quality in the data warehouse versus real-time data quality in the call centre. Separate tools for each of these instances would almost certainly lead to different standards, duplicate values and other inconsistencies between the transactional application and the data warehouse. This, in turn, will perpetuate the mis-alignment and lack of understanding that is so common across different business functions, and between those people who are making strategic decisions as opposed to those that are actually executing day-to-day processes. The remedy for such a mismatch is to look for a data quality platform that extends across the enterprise both for different types of applications and different types of implementation.

If you accept the idea that the most desirable approach will be to have a single data quality platform you might then ask whether this should be a part of a larger data integration platform or whether your needs will be better served by a data quality specialist such as Trillium.

Specifically, of course, Trillium aims (and claims) to provide more functionality than its competitors, especially those coming from the ETL space. Hence the introduction of phrase analysis, the range and extent of Trilliums interfaces (it aims to integrate with anything), its new API, its Unicode support, its TS Insight product for trending and reporting, and so on.

However, leaving aside these technical considerations there is a clear argument for investing in an integrated ETL/data quality environment: one vendor, one tool set and so on. However, there is also a clear argument to not invest in this way, because many data quality requirements have nothing to do with data integration or ETL. If you want to embed data matching, say, in a call centre application then this has nothing to do with ETL, so why use an ETL-based tool rather than a pure data quality offering?

To take this further: the fundamental purpose of IT is to provide information to the business in a suitable format and in a timely manner; and data quality is fundamental to realising the value of that information. However, most large organisations have a wide variety of technologies in production and have very rapidly evolving business requirements. Because of this, data quality processes and capabilities must be deployed in a wide variety of contexts: batch and real-time, at the point of data capture and the point of data extraction, in data migrations and in on-going database maintenance operations. A data quality platform that can be directly integrated within these various contexts is likely to be more flexible and scalable (quicker to deploy and cheaper to operate) than data quality components that are wrapped within a specific environments such as a data integration suite. If you buy into this argument then Trillium must be a leading contender for any such platform.

PLEASE NOTE: This is an editorially-independent, sponsored article. If you would like your company, products or solutions to be featured on IT-Analysis.com please contact us.