In October 2012, the Harvard Business Review named “Data Scientist” the “sexiest job of 21st century.” While this profession has since gained meaningful acceptance and understanding in the broader big data analytics world, the dearth of real data scientists (some believe there are only 3,000 in the world), has opened the door to what I call “data doctors.” And many enterprises desperate to make sense of their burgeoning data to drive business value might get duped by IT consultants or aspiring candidates masquerading as data scientists.
What’s the distinction between the two? Data doctors – business intelligence analysts, ETL developers, data assemblers, data quality testers, data analyst, etc – are skilled in working with data. But data scientists typically deal with complex algorithms and statistics to unearth the hidden treasure in big data. They give yet unexplored meaning to the data. Their work has a higher degree of risk and probability to fail, but it also delivers the highest rewards. They are the ones responsible for big-ticket transformational ideas.
Yet, with the increasing consumerization of analytics and the realization that the data scientist pool is minute, many enterprises believe they do not need data scientists as:
- There may not be all that much value in the data
- Data scientists are very expensive
- When given the right tools, data doctors can replace data scientists.
While it’s true that data scientists are expensive, the other two above points are erroneous. There is a lot of value in data that data doctors are unable to mine. And assuming that a college graduate or an IT engineer adept in BI technologies can become or substitute for a data scientist by leveraging new age big data solutions is a mistake.
These “consumer focused” solutions hide the complexities of generating meaningful insights, data discovery, and visualizations by adopting a WYSIWYG (What-You-See-Is-What-You-Get) approach where users can assemble workflows and analytics logic/model using drag and drop in a highly intuitive user interface. These technologies are destroying the data custodian ivory towers of corporate IT, and making business analysts perform substantial analytics projects on their own. But make no mistake… they do not reshape analysts, the data doctors, as data scientists.
While it is true that new age technologies help data and business analysts skill up to perform more advanced analytics, assuming this eliminates the need for data scientists is akin to saying we don’t need human pilots due to the auto pilot function. Indeed, the great demand for real data scientists is the reason many universities have launched dedicated data scientist programs (e.g., advanced analytics programs at Columbia’s Institute for Data Sciences and Engineering).
The real value of these new analytics tools is in enabling data doctors to perform many tasks previously handled by data scientists, thereby freeing the prized scientists to work on resolutions to highly complex problems that can significantly benefit the business. They also help enterprises who believe data scientists are overkill to enable data doctors to perform reasonably complex tasks.
However, enterprises really interested in data-driven insights will be best served by empowering both scientists and doctors. The doctors need to keep updating their knowledge about the latest analytics solutions that can help them add more strategic value. And data scientists need to dive deep, unravel unexplored territories, and develop data-driven insights that can transcend the boundary of human intelligence.
Photo credit: Stephen Coles
