We have moved from an era in IT where the relational database management system was the one-size-fits-all technology, where data being used for informational purposes was solely sourced from in-house corporate databases that were rigorously defined, structured and well within the span of control of IT. 

In the post relational era, new types of database management systems designed to handle data previously untapped for informational purposes because of volume, velocity or its inherently unstructured nature are coming online. The Big Data and NoSQL databases are challenging the dominance the more traditional relational database management systems. 

At the same time, the rise of so-called shadow IT systems developed largely outside the span of control of IT and often implemented in the cloud are being rolled out at an unprecedented rate. The data from these systems is making its way into enterprise data warehouses and data marts to be used for informational and decision making purposes.

Taken together this means that data being used for decision making is no longer limited to what can be sourced from corporate systems of record and that the new sources of data are coming online and changing at an ever more rapid pace.

All of this is happening now when most corporations barely have a handle on the data contained within their corporate systems of record. Data governance and the practice of data management is seen to move at sluggish pace, focused on the notion that there is a single version of truth to be discerned from corporate data. 

To meet this need, data governance itself must evolve. It must move away from the idea that only data deemed as a single version of truth can be used for informational purposes. It must become more agile as a practice adapting at the pace of change that exists today in IT. Finally, it must invest more time, in the definition of the kinds of metadata needed to provide context to relatively unstructured data. 

Today the governance of data can no longer driven solely by the idea of a single version of truth. Data needs to be evaluated based on different levels of trust, privacy, timeliness, confidentiality and so on, so that a profile that can be developed and associated with each data set. This profile indicates the inherent level of risk and value to using this data. 

Data governance and the practice of data management needs to keep pace with the rate of a change in new sources of data.  When data was solely sourced from corporate systems of record, the structure of that data was well defined and relatively static once in production. Changing the structure of a customer record for example required careful consideration because of its impact on a myriad and growing number of corporate systems that relied upon it.

While this remains true of corporate systems of record, the structure of data from shadow IT systems, and external data sources such as social media are much more likely to change and grow over time. The practice of data management needs to become as agile, as the development of new systems has become. 

New kinds of metadata need to be defined, providing much needed context to relatively less structured data. This meta data needs to be made readily available and accessible to more sophisticated kinds of end users of the data. The business value of: Data as a Service, of fostering user self-sufficiency, and of monetising approaches to data, is closely linked to the quality of the metadata that describes the data. 

Data governance is what drives business value and mitigates the risk of using data but it cannot be a roadblock to change, it must evolve and mature to keep pace with a rapidly growing number of new kinds of data sources. 

Dennis Layton is an Enterprise Data Architect, based in Canada with over 35 years of IT experience. He is an advocate for a more agile and adaptive approach to data management.