Data engineering is one of the fastest growing data disciplines. It’s actually been around for a long time behind the scenes but only recently getting the credit it rightly deserves.
Why do you need Data Engineering?
The following scenarios happen fairly often:
- Company X invests tens of millions in a Data Platform with Artificial Intelligence and Machine Learning capabilities which is launched with great fanfare. However, AI/ML fails to deliver any real and meaningful output because the data that feeds the platform is from limited sources and of poor standard.
- Organisation Y hires a large Data Science team with the promise of focussing on data science to transform the business but the data scientists leave because they do more data wrangling than data science as there’s no data infrastructure in place.
- Company Z launches innovation projects that demonstrate value but when loaded with significant amounts of data, from multiple sources, of different standards and quality, which is constantly changing, the innovation projects fail because they are “proof of concepts” and aren’t able to scale. The C-Suite loses confidence and does not invest further in an enterprise-wide capability.
All three scenarios could have been different with a Data Engineering capability.
What is Data Engineering?
Data engineering is about the data infrastructure and data quality that feeds data analytics and data science. Data engineers build and maintain the systems and structures that collect, extract, store, organise, and protect data. The best data engineers have a deep understanding of the data fundamentals that they bring to the many bespoke and complex data problems that organisations face that can’t simply be fixed with automated SaaS platforms or tools.
At Envitia, we have a data engineering capability based on years of deep data applied research that has developed expertise in the core components of data engineering, so our data engineers have that deep understanding. We believe that whilst many data tools on the market have their place for the specific purpose for which they have been designed, and we do use many of them, simply being skilled at using that tool does not make someone a data expert.
We organise data engineering into three services.
- Data Modelling – Fundamentally any piece of data is a representation of a real-world measurement. Data modelling is the way we represent that real world in a digital form. Modelling to the right level means that a users perspective of that data is intuitive and easy-to-use, to answer those questions that an organisation’s business needs.
Importantly, a good data model is designed with the future in mind compared to an “ad hoc” data application for an immediate purpose, such as the innovation projects mentioned above. It will “flex” with an organisation’s changing data needs such as an explosion in an organisation’s data, from internal as well as external sources, increased complexity of data, and different data types and formats. This is simply because a data model is about storing the data in the right way, and capturing the relationships between that data, which is where future insight can be derived.
- Data Quality – Data is always uncertain. The key is knowing where that uncertainty is and what risks it brings to the underlying decision-making that it drives. Our data quality service assesses and rectifies underlying data quality issues to unlock the true value of the data. Quality is measured in a number of ways and we score and correct data across “dimensions” to generate an overall data quality assessment. We consider accuracy, completeness, consistency, timeliness, validity and uniqueness as these all play their part in overall quality thus determining what the data can be used for.
- Data Warehousing/Cataloguing – After we have worked out how to store the data (modelled it), brought it up to the right level (of quality), we then make sure it’s useable across the organisation. This service is where we build a data platform, which can be data catalogues, warehouses, lakes, or simple repositories where the data lives. We build data pipelines and get the data into the platform where it can be discovered, analysed and acted upon by users to drive business decisions. We pride ourselves on being technology agnostic to provide the right solution not just the latest technology.
All three of these above services deliver a well-designed, workable, adaptable data solution, capable of scale, that delivers good quality data to a data science capability. Thereby delivering the value that C-Suite require.
Further blogs to come on each service soon.
How to start a data engineering journey
We always start our customer discussions with Data Architecture, covered in a separate blog Reinventing Enterprise Architecture to Deliver Digital Transformation Success, by making sure we focus on the business challenges and understand what data could/should do for an organisation. This means we can design and implement data pipelines, using the data engineering components above, fit-for-purpose today but also fit for future use-cases and enabled for scale.
When done right, no-one notices data engineers because the right data is delivered at the right time, at the right data quality. We’re proud to keep it that way.
Please contact Sales@Envitia.com so we can understand more about your data needs but more importantly what you’re aiming to achieve.
By Richard Griffith
CTO, Envitia
