Analytics in the Enterprise Survey

How Data Science and Analytics are Being Used WithIn Organizations

By Tony Ojeda
Last Updated: June 25, 2019

A few months ago, we sent out a survey to get a better sense of how data science and analytics were being used within organizations. Several people from our professional community responded, and now that we’ve had a chance to analyze the responses, we’ve decided to publish the results here so that anyone interested could see them.

Note: If you’d like to complete the survey, you can do so here. We will be updating the results on this page periodically, so make sure to come back and see how things have changed.


REspondent and Organization Information

The first part of the survey was intended to capture some general information about the respondent and their organization, including the size of their organization, their industry, and their functional department.

What is the size of your organization in terms of number of employees?

Most of the responses we received (72.5%) were from people that worked for either large companies or small companies, with medium-sized firms making up the remaining 27.5%.

organization size

What is your organization’s industry?

Since District Data Labs was founded and grew out of the data community in the Washington D.C. area, a significant percentage of the survey responses received were from people that worked in either the Software & Internet industry or in Government (39.3% combined). Business & Professional Services, Education, and Healthcare & Pharmaceutical were the next most common industries, representing just under 10% of respondents each.

industry.png

What functional department do you work in?

For similar reasons, the most common functional roles of respondents were those in Analytics (32%), IT & Software Development (26%), and Research & Development (13%).

functional.png

Data Science and Analytics Within Organizations

With the second part of the survey, we wanted to gain some insights into how data science and analytics are currently being applied. To obtain this information, we asked respondents a series of questions about the analytical practices, methods, metrics, and tools used within their organizations. Below are the responses we received.

Does your organization have basic reporting in place for monitoring performance?

Reporting is perhaps the most critical and foundational analytical practice that an organization can have. It allows management and employees to see how well the organization is performing and provides visibility to issues that may need attention or resolution. Most respondents reported that their organizations do have basic reporting in place for monitoring performance. However, almost 16% reported that their organizations did not, which is a bit surprising given how important it is to an organization’s management.

reporting.png

What key performance indicators (KPIs) or metrics are most important in your organization?

When asked what key performance indicators (KPIs) are most important in their organization, respondents cited metrics having to do with revenue, employee performance, customer/user engagement and retention, products, growth, and project success. Below is a word cloud visualization showing the most popular terms from the responses to this question.

kpis.png

Are you able to extract data from your organization’s systems?

When embarking on a data science initiative, one of the first obstacles you are likely to encounter is a limitation or constraint that prevents you from being able to access the data you need. In some organizations, data is siloed or locked up in the systems the organization uses to run their operations, and we wanted to get a better sense of how prevalent the problem of not being able to access necessary data was. From the responses, it seems as though most respondents (84.1%) are able to extract the data they need from their organization’s systems.

extract.png

Does your organization have a data warehouse or other centralized repository where data is stored?

In addition to being able to extract data from your organization’s systems, it is also very helpful to have a centralized data repository where data is cleaned and properly structured for conducting analyses. When asked whether their organization had such a centralized repository, 65.2% of respondents said that their organizations did, 23.2% reported that their organizations did not, and 11.6% said they were not sure.

repository.png

Does your organization use any business intelligence software?

For many organizations, business intelligence software serves the purpose of democratizing analytics by providing access to critical reporting and insights to a wider audience than just data scientists and analysts. According to respondents, 58% of their organizations had some sort of BI software implementation.

bi software.png

What business intelligence software does your company use?

Among the respondents who reported that their organizations did use business intelligence software,Tableau was the most popular BI tool with 26% of respondent organizations using it. Coming in second was Microsoft’s PowerBI (18.8%), followed by a tie between Qlik and Alteryx for third place with 6% each.

bi_software_usage.png

Does your organization have any data scientists and/or data engineers on staff?

We often hear in the news and anecdotally that companies are increasingly getting on the data science bandwagon and hiring data scientists and data engineers, so we wanted to get a better sense of what percentage of organizations now have data scientists or data engineers on staff. As the chart below shows, 77.6% of respondents reported that their organization did have data scientists or data engineers working for them.

data scientists.png

Does your organization have a dedicated data science or analytics team?

In addition to wanting to know whether their organization had data scientists or data engineers on staff, we also wanted to know what percentage of organizations had dedicated data science or analytics teams. Whereas hiring individual data scientists or engineers shows that a company values their data and wants to extract insights from it, having a dedicated data science team implies a deeper commitment to analytics and the impact it can have on their operations. Overall, respondents reported that 62.3% of their organizations had a dedicated data science or analytics team.

data team.png

Which analytics tools are used in your organization?

There are a number of debates online about what tools are best for data science (ex. R vs. Python), so we wanted to include a question about what tools are used to perform analytical tasks within the respondents’ organizations. From the list of tools we provided, Python was the most popular with almost 70% of respondents reporting that it was used for analytics within their organization. Excel came in a close second at 62%, and R came in third at 50%.

analytics tools.png

Does your organization use predictive analytics or machine learning for forecasting important metrics?

Just over half (52.2%) of respondents reported that their organization leverages predictive analytics or machine learning to improve their forecasts. When asked a follow-up question about what metrics their organization uses predictive analytics or machine learning to forecast, respondents cited top-line sales and revenue figures as well as user and customer-centric metrics such as clicks, conversions, retention, responsiveness, and engagement.

predictive analytics.png
machine learning metrics.png

Does your organization use deep learning or artificial intelligence to perform tasks requiring human-level intelligence?

The next question in the survey aimed to get a sense of the level at which deep learning and artificial intelligence methods were being used by organizations to perform tasks requiring human-level intelligence. Despite all the hype around these methods in the news these days, respondents reported that only 34.8% of their organizations were using these methods. When asked a follow-up question about what tasks their organization uses these methods for, respondents cited automated analyses and workflows as well as scientific research, image recognition, and natural language processing applications.

ai deep learning.png
ai tasks.png

On a scale from 0 to 10, how strong do you feel your organization's analytical capabilities are?

We also thought it would be interesting to measure how strong respondents felt their organizations’ analytical capabilities were. The ratings respondents gave their organizations most frequently were 6 (18.8%), 7 (14.5%), and 4 (13%). Additionally, 23.3% of respondents gave their organization’s analytical capabilities a score of 3 or less, and another 23.2% gave their organization a rating of 8 or higher. When asked what they felt their organization’s biggest analytics challenge was, the responses ranged from limitations and constraints around collecting data, competing priorities, internal processes, and availability of appropriate analytical tools.

analytical capabilities.png
analytics challenge.png

Does your organization track customer metrics such as acquisition rate, retention rate, and customer lifetime value?

Next, we decided to drill down into more specific types of analyses to see how common each type was among organizations. The first one we asked about was customer metrics such as acquisition rate, retention rate, and lifetime value. According to respondents, only 53.5% of their organizations are currently utilizing these types of customer analytics.

customer metrics.png

Does your organization segment customers into different groups based on common attributes or behaviors?

Another useful application of analytics to business is segmentation of customers into groups so that they can be marketed to, served, and assisted more effectively. According to respondents, 63.8% of their organizations engage in some sort of customer segmentation analysis.

customer segmentation.png

Does your organization leverage analytics to predict customer propensity (likelihood of purchase) and capacity (amount of spend)?

A particularly useful application of predictive analytics and machine learning to business is customer propensity and capacity modeling, where propensity measures the likelihood that a customer will make a purchase and capacity attempts to measure the amount of money they are likely to spend. When asked, our respondents reported that only 44.9% of their organizations calculate these metrics for their customers. This is an area of opportunity, as this type of modeling can help an organization maximize the effectiveness of their marketing dollars by focusing on the customers that are most likely to buy and have the most money to spend.

propensity capacity.png

Does your organization track the profitability of individual products/services and product/service groups?

In addition to customer analytics, data science methods can also be valuable when used to analyze the products and services an organization sells. One of the most basic types of analysis is calculating the profitability of each product or service as well as the different groups or categories that those products/services are classified into. According to our respondents, 55.1% of their organizations track this, which was a bit lower than we expected given the potential benefits to an organization of identifying which losses to cut and which winners to let ride.

profitablility.png

Does your organization calculate how likely sales volume is to increase or decrease at varying price points for products/services?

The next question we asked had to do with analyzing price sensitivity or how likely sales volume is to change in response to increases or decreases in product and service pricing. Performing this type of analysis can help a business gain a better understanding of what the optimal trade-off of price vs. quantity is for each of their products and services. However, only 42% of respondents reported that their organization currently performs this type of analysis, which means this could be another area of opportunity for these organizations.

price sensitivity.png

Does your organization personalize products/services for customers?

As industries get more competitive and customers have more vendors and offerings to choose from, personalization of products and services becomes increasingly important. Because of this, we wanted to get a better sense of what percent of organizations personalize their products and services for their customers. According to respondents, 52.2% of their organizations perform some sort of customization of offerings for their customers.

personalization.png

Does your organization have an automated way of recommending to customers products/services that may be a good fit for them?

A good way to personalize offerings for customers at scale is by leveraging the power of recommender systems. Recommender systems analyze information about customers and their past purchases, compare their behavior to that of other customers, and then recommend products or services to them that they are likely to want. Despite the value recommender systems can deliver, only 30.4% of respondents reported that their organizations have an automated way of recommending products and services to customers. This implies that the majority of the personalization efforts are being performed manually.

recommender system.png

On a scale from 0 to 10, how efficient do you feel the processes at your organization are?

Speaking of manual processes, another benefit of data science methods is automation of time-consuming and information-intensive tasks. To get a better sense of organizational efficiency among our respondents’ organizations, we asked them to rate how efficient they felt processes were at their organization. The vast majority of respondents gave their organizations a rating between 3 and 7 (78.2% combined). Outside of that range, 4.3% of respondents gave their organizations’ processes a rating of 2 or lower and 17.5% rated their organization an 8 or higher.

process efficiency.png

Does your organization have processes that rely on someone manually pulling information from data sources to put together reports or make decisions?

Getting a bit more specific, we wanted to know whether their organization had any processes that relied on manually pulling data for reporting or decision-making. The majority of respondents indicated that they did (73.9%).

manual data extraction.png

On average, how long would you say it takes to get the data needed for reporting or decision-making?

We followed the previous question with one that asked how long it usually takes to get data at their organization. The most frequent response for this question was between 3-5 hours (24.6%), followed by a tie between 1-2 hours and 1-2 days (15.9% each).

data wait time.png

Conclusion

Overall, we found the results of this survey quite interesting. The responses to some questions were in line with our expectations while other results were a somewhat surprising. At the end of the day, these results will help us better understand the people and organizations within our professional community and develop training and consulting offerings that are genuinely useful to them.

As mentioned at the beginning of the post, we plan on updating these results periodically, so if you’d like to help us out and take the survey, you can do so here. Thanks in advance, and stay tuned as we continue extracting additional insights from future responses!


Does Your Organization Need Data Science assistance?

Check out our services to find out how we can help!