Data estate complexity is a big challenge for decision makers

A new report from Capital One and Forrester also finds that respondents have difficulty governing data at scale overall.

At a time when organizations are increasingly adopting a cloud-first approach and global spending for big data services is expected to surge to $273 billion in 2026, 82% of data management decision-makers are deeply concerned about controlling and forecasting rising data costs, a new study finds.

Most concerning is that one of the biggest data challenges decision makers currently face is fully understanding the complexity of their organizations’ data estates, according to the report from Capital One and Forrester. And nearly three-quarters of data management decision-makers have yet to manage most of the data their organizations have in the cloud, the study found.

More than half of respondent decision-makers (56%) noted their organizations maintain a centralized data estate that stitches most or all organizational data. However, a small percentage (15%) are federated, which means there is a central set of data policies and platforms, but lines of business manage data independently through self-service, the report said.

“A federated model supports the inevitable expansion of a data estate as data volumes, sources, and tools multiply, eliminating potential bottlenecks in the enterprise. However, while federating and enabling self-service, the data estate must still remain well-managed,’’ the report said.

Among the other findings:

• Talent challenges are the biggest barriers to data management excellence. Decision-makers do not have enough staff, in-house expertise or collaboration between teams and tools to achieve their outcomes.
• Data usage costs are increasingly unpredictable, with 82% of data management decision-makers citing difficulty forecasting and controlling costs as a top data ecosystem challenge.
• Nearly 80% of decision-makers cited a lack of data cataloging as a top challenge. Without data cataloging, decision-makers struggle to understand what data they have, how the data is used and who owns it.
• Other key concerns included poor data quality (80%), difficulty understanding data (76%) and a lack of data observability (74%).

SEE: Hiring Kit: Database engineer (TechRepublic Premium)

Tracking and enforcing data governance policies is another concern

Data governance is critical to business success because it “makes data relevant and ready for analytics, insight, and action,” the report noted. “If users can’t access data, they can’t blend it; if the data is of poor quality and/or does not suit their needs or area of responsibility, they won’t use it.”

Yet a top challenge for data management decision-makers is tracking and enforcing adherence to data governance policies. Only 20% claim that they have no difficulty governing data at scale. Eighty-two percent noted confusing data governance policies and 80% noted a lack of entitlements and enforcing role-based access to specific data as top challenges.

Data management capabilities are expected to drive spending

Decision-maker respondents anticipate that investment in entitlements is expected to nearly double from 17% today to 31% in one to two years, according to the report.

Nearly half of the respondents said cloud data investment priorities going forward will include protection and security (49%), tools that make data easier to find and use (47%), along with tools that reduce and manage costs (39%).

Multiple providers will be used for different use cases

Respondent data management decision-makers understand partners are critical to helping them manage their organizations’ data. Eighty-seven percent of those surveyed are using commercial tools, while just over a quarter (28%) use a mix of homegrown or open source tools.

To create a more diverse data estate, organizations must get their data from a variety of sources. Over the next two years, the majority of decision-makers (84%) plan to use multiple providers or different providers for different use cases.

“This is in stark contrast to decades prior where just a single provider delivered most functionality necessary to manage data estates,’’ the report noted. “Today, only 10% of respondents use different vendors for different use cases. Over the next one to two years, multivendor use will increase nearly four-fold to 38%.”

SEE: Microsoft Power Platform: What you need to know about it (free PDF) (TechRepublic)

Consider tools that provide visibility into data costs

As organizations progress on their cloud journey, they must adjust how they balance global and local data use with a federated approach to ensure governance requirements are met and data access, quality and observability are at their best, the study suggested.

Organizations must invest in tools that control and forecast data costs. Inherent exponential costs associated with operating on the cloud need to be addressed early to ensure success, the report advised.

Additionally, “as organizations progress on their cloud journey, they must adjust how they balance global and local data use with a federated approach to ensure governance requirements are met” to ensure the best data access, quality, and observability, the report said.

Capital One commissioned this report and Forrester Consulting supplemented this research with custom survey questions asked of 157 data management decision-makers in North America. The survey was conducted in March 2022.