I’ve been a market and product analyst for large companies. I realize that my experiences are a sample of one, and that I can’t speak for my analyst peers. But I suspect some of them would nod in recognition when I say that in those roles, I spent only a fraction of my time in these analyst roles actually conducting data analysis. With the increase in press that Big Data has received, I started seeing a major gap between what I was reading about enterprise data trends, and my actual experiences working with enterprise data.
A more accurate description of what I spent large amounts of time doing was data hunting. And data gathering, and data cleaning, and data organizing, and data checking. I spent many hours trying to find the right people in various departments who “owned” different data sources. I then had to get locate definitions (if they existed – this was hit or miss) and find out what quirks the data had so I could clean it without losing records (for example, which of the many data fields with the word “revenue” in it would actually give me revenue). In several cases I found myself begging fellow overworked colleagues to please, please, pull the data I needed from that database which I in theory should have had access to but was shut out of due to multiple layers of bureaucracy and overall cluelessness as to what data lived where within the organization.
Part of me thought, “Well, this is the lot of an analyst in a large company. It is the job.” And this was confirmed by other more senior managers – all on the business side, not in the IT side – who asserted that, yes, being a data hunter/gatherer/cleaner/organizer/checker was indeed my job. But another part of me was thinking, “These are all necessary tasks in dealing with data. I will always need to clean data no matter what. I will need to do some formatting and re-checking to make sure what I have is correct. But should this be taking up such a large chunk of my time? This is not the best way I can add value here. There are too many business questions I could potentially be trying to help solve; there has got to be a better way.”
So initially I thought, not being an IT professional, that this was an issue of not having the right IT tools. But gradually I came to understand that technology was not the problem. More often than not, I had access to best-in-class CRM systems, database and analytics software, and collaboration tools at my disposal. I had the latest versions of Microsoft Office and a laptop or desktop with decent processing power. I had reliable VPN connectivity when I was working remotely and often a company-supplied mobile smartphone. It was the processes and people that were the biggest barriers to getting the information I needed in order to provide fact-based research that could be used to solve business-critical decisions.
Out of sheer frustration, I started doing some research to see if there was indeed a better way for enterprises to manage their data. Master Data Management (MDM), you’ve been around for over a decade, why haven’t I ever encountered you? A firm called the Information Difference, a UK-based consultancy which specializes in MDM, argues that too often, decisions about data management and data governance are left solely to the IT department. The business should also be part of any MDM project, and the governance process should be sponsored and led by C-level business management. Talk about “aha” moments. When I read this, I actually breathed a sigh of relief. It isn’t just me that thinks there has to be a better way to go, so that the not-cheap business and market analysts that enterprises the world over employ can actually spend more of their time solving problems and less time data wrangling!
That’s why when I read the umpteenth article/blog post/tweet about how transformative Big Data is and will be, I cannot help but groan. Before enterprises begin to think about new ways about structuring and distributing data, they need to do an audit of how existing data is already used within and between different businesses. In particular, they should consider MDM if that has not already been implemented. There is so much valuable data that already exists in the enterprise, but the business and IT have to actually work together to deploy and communicate about data initiatives. They also need to evaluate if and how enterprise data is being used effectively for business decisions, and if that usage meets compliance and security rules.
I suspect that many senior IT managers know this and agree. I also suspect that getting counterparts in the business to be active and own decisions about enterprise data, and not just think data is an IT issue, can be a challenge. But in the long run, if this doesn’t happen more often, there’s going to be a lot of overpaid, underutilized data analysts out there and missed business opportunities. So if you are an enterprise executive wondering “do I have to worry about this Big Data business?” please take a step back and look at what you already have. And if you know any seasoned data analysts in your company, maybe even talk to them about what would make them more effective and faster at their job. The answer may be simpler than you think.
