No consulting project has perfect data.  In fact, it is usually a little bit like an Easter egg hunt where the team has a good idea where the data eggs might be, but can’t be 100% sure until they really start looking for it.

Most companies have trouble with their data.  While there is a lot of talk of BIG DATA and the revolution it will have in predictive analytics, in reality, many organizations have trouble patching together their SMALL DATA.  I would guess that only 1/3 of companies can can satisfy the consulting team’s data request.

Oddly, lack of data = consulting project.  For a bit of circular logic. . . it is often because the data is difficult to find, that the client has not really solved the problem yet.  As Seth Godin remarked in a blog post about “perfect problems”:

The only problems you have left are the perfect ones. The imperfect ones, the ones with a clearly evident solution, well, if they were important, you’ve solved them already

So, as odd as it sounds. . . consultants need to be thankful that the client’s data is usually a mess.  It means more work, projects, billing and money.

Data collection is often painfully slow.  Even when the location of the data is clear, it is common for the team to spend several days hunting down the right people to get the data.  Consultants often go to the client site just to request (read: pester, nag) the client to give them the data.  Clients could save themselves 5-10% of fees if they would just get the data to the consultants quicker.

Good data is hard to find.  In my experience, the larger, the more geographically dispersed, and the older the company = the messier the data.  Using the analogy of data flow like plumbing . . . the larger and older the house, the more it leaks.

#1-3. IT needs to update and standardize.  Too often IT only makes tactical repairs and spends their energy and budget just playing catch up.   Too often, clients customize the enterprise resource planning (e.g., SAP, Oracle) to match their process, instead of listening to the systems integrators and stick with best practices.  “Oh, we like to do it our way” is usually code words for messy data down the road.

#4. Sometimes legacy = bad.  Too often there is a resigned exhaustion of existing legacy processes.  “Yes, we do it on paper because we always did it that way.”   “No, I don’t have it written down because I remember it all.”  Those are all signs of trouble and poor planning. There is always someone – often with white hair and a beard – who knows how things really work. Talk about disaster planning: What happens if Al leaves?

#5-7. Time to clean the data.  The customer master (where you list all the key information of your clients) needs to be clean because it is used for billing, accounting, and other customer-relationship activities (e.g., sales calls, marketing direct mail).  Too often these are a bit of a mess.  For example, there will be 4 ways to spell Wal-mart, Wallmart, Walmart, Wall Mart.  With junky data, it is hard to analyze anything.

#8-9. Figure out the roles / responsibilities.  Who’s job is it anyways?  If it is everyone’s job, then in effect, it is no one’s job.  It’s not a good sign when veteran office workers who are uncomfortable using basic excel commands like sort and pivot.  While it seems basic, sometimes real-time analysis is just not valued enough to put into job descriptions and performance reviews.  Inevitably, it is the managers fault.

Data is not just lying around.  For those new to consulting, get ready to start digging for the data. Just some of the crazy examples from my past:

• On my first consulting project, I spend several late nights alone typing shipping data from paper invoices into an excel spreadsheet
• Just a few months ago, we consolidated data from 60+ separate emails into one excel.  How can you look at trend data when it is sitting in 60 daily reports?

Ask any analyst, and they will have their own hazing story of collecting data in some manual and crazy way.  As long as companies don’t take the time or the effort to do this, they will continue to pay $$$$ / hour for this mundane task to get done.

Data is a bit of a misnomer because it treats everything the same.  In my mind, there is a progression / hierarchy of the value of information which looks something like this.  It starts as noise, gets organized into data.  That data turns into information as it is structured, cleaned, rearranged, and sorted so it makes some sense.  Analysis takes shape as information is pivoted, correlated, appended, hypothesis-tested.  Insights are really gems and diamonds.  They are rare, valuable, and often very polished.  Only analysis and insights should be presented.

Lack of data requires consultant creativity. Sometimes, consultants have to uncover, create, cleanse, triangulate or even create data to answer key questions.  Creativity is needed here.  It is also a great way to “wow” the client.

Creating data is not sketchy or unethical. Some of consulting tools used to find new data include: surveys, interviews, focus groups, workshops, financial comparisons, observations, estimates, simulations, business models, benchmarks, maturity models, and others.  More on surveys in the next post.

Please feel free to add any data war stories you might have.