Design Considerations for Building Data Centric Platforms
Business Drivers
Business value for enterprise applications essentially depends on the wealth of data that can be leveraged. Inaccurate or inconsistent data can result in incorrect business decisions, improper revenue recognition, introduction of new risks, low effectiveness in marketing campaigns and delays in time to market.
Data is the essential asset and is shared across multiple systems in every enterprise to run business operations. Since it can be the primary product of the organization bringing in the revenues, it deserves due attention.
Data Collection & Ingestion
Data required for business processes might not be completely owned and produced by the organization. Thus, every organization needs to build the Data Integration capabilities that include collection and ingestion of data.
- This is the discovery of ‘What’ data is needed. Common language between business teams to draw the comprehensive list of required data items is essential
- ‘Where’ – refers to the source of data. Each data item should be mapped to a source which could be internal to the organization or external, and qualifiers need to be defined to identify the correct source. Few cases would necessitate identifying more than one source for the same data item for validity and correctness
- ‘When’ – each data item in the real-time would have a shelf life and a frequency of update, and updating whole or part of the data needs to be identified and planned for
- ‘How’ – this refers to the Extract, Transform, and Load (ETL) process of constructing the data platform. Data could be shared as flat files, SQL query from a DB or a network stream. Each data source has to be evaluated to identify the optimal way of obtaining the data. The extracted data then needs to be transformed to fit the desired format and loaded in to the target locations. Suitable tools and technologies such as Informatica, IBM Infosphere Information server, Oracle – Data Integrator, etc. for extracting data from variant sources have to be evaluated and finalized
Data Quality
The data-centric platform has to offer uncompromised quality for businesses to operate successfully. Quality can’t be achieved if it’s not addressed from the beginning. Identifying correct data sources, especially external sources involve evaluating with respect to the historical performance; time bound validity of the information, format and consistency.
- Few strategies like identifying two or more sources of the same information and considering the value repeated for maximum times have to be incorporated to collect the reliable data
- Data Type validations, format validations, cleaning unwanted characters, completeness of data are all part of quality checking while extracting and loading the data
- Strategies to lay out the error detection and fall back mechanism in case of failures in extracting and loading is important for data integrity
- Logging the failure occurrences, auditing and monitoring the performance of each data source helps to qualify the providers and builds the reliable set of vendors
Data Latency
Primary characteristic of the data platform, which is the Master Data for all business operations, is the Single Point of Truth for all kinds of data elements. It is quite often that data about the same element is stored in different systems in various formats. In few instances, if the updates are not cascaded on time, it could leave scope for inconsistent data. Consolidating and normalizing information so that the data platform is the only source of all information is essential. Building the Data Virtualization Layer to be the single window of serving the data helps to use the same version consistently across all processes.
Data Security
Building the data platform is not complete unless security is implemented. ‘Security’ refers to protecting data from unwanted access, corruption and destruction.
- Hardware security strategies like password locked hard disks, biometric based access to data systems can be opted as they are less vulnerable when compared to software only security mechanisms. Primary level software security is also essential to prevent easy access to the data
- Protecting vital company information in non-production environment has become critical. Production data replicated in development and QA environments is vulnerable for data theft and insider exposure. Data encryption might cause performance degrade making data masking a better alternative. Implementing masking starts with identifying sensitive data elements, the tables and relationships specific to the organization, and identifying masking algorithms.
Business Intelligence
Business Intelligence (BI) applications use the integrated, consolidated and cleansed data to provide a broad set of data analytics such as ad-hoc, statistical and predictive analytics. Rich data visualization applications depicting performance scorecards, business metrics, key performance indicators, graphs, charts and enterprise reports help the business to improve the operational efficiency and generate new revenues.
Presenting analytics, dashboards and reports is facilitated by BI experts and data scientists in accordance with the business users’ needs. This may cause a bottleneck in the process and limit the business user’s freedom to go beyond the provided set of analytics. Thus, self-service or DIY intelligence tools are becoming common allowing end users to run custom queries and reports to derive data-driven insights to use in their course of work.
The effort of building and maintaining the Data Centric Platform is justified when the data is transformed into a meaningful information asset which can drive strategic and tactical decision-making process.
