Moving Beyond BI Shadow Systems

A recent post on The Data Doghouse, a Business Intelligence blog, explains that data shadow systems are still extremely prevalent across industries and will continue to be so for the foreseeable future. Data shadow systems are groups of spreadsheets and local, customized databases – often built in Microsoft Access or Excel – created by business groups to provide data for their users. An excerpt from the article shows just how prevalent these shadow systems are:

In a survey we conducted in the fall of 2007 we found that the medium number of data shadow systems at an enterprise was 30. According to the survey, data shadow systems are prevalent in all industries, companies of all sizes and throughout various business functions in the enterprise.

The presence of shadow systems is a trend we’ve seen with our own clients in education, government, banking, and insurance. For many of our clients, different shadow systems are providing business analysts with the data they need to answer important business questions, recognize trends, and make informed decisions. Obviously, these shadow systems are providing a valuable service. However, despite these benefits there is a downside to operating shadow systems. Let’s take a look at some of the negative aspects of shadow systems.

First, it is often time consuming to implement and maintain data shadow systems. Data shadow systems are typically cobbled together by business users. Let’s face it, designing these systems is not the strength of business users (this is something that is bettered handled by the IT department). This leads to a design framework that requires lots of labor-intensive maintenance activities and a lack of felxibility and extensibility.

Second and most importantly, these shadow systems do not enforce rigorous data management techniques and principles. For example, consistent business rules may not be used across the organization to clean and codify the same data. The result, data quality and consistency suffer and it is difficult to know how confident decision-makers can be in the numbers being reported.

So the question is, what can be done to move beyond the use of shadow systems in your organization. The best approach is to engage the IT department and use the organization’s existing Business Intelligence (BI) tools to tackle this problem. BI tools should be used to centralize data, implement consistent and documented business rules to clean data, and develop reports that meet the needs of business users and stakeholders.