Get weekly insights on modern data delivered to your inbox, straight from our hand-picked curations!
This is a follow-up piece which addresses the feedback we received on how we can enhance the end-to-end Data Product Guide.
Part 1 addressed:
“An easy way to convey the value of the lifecycle to stakeholders who aren't invested in detailed materials or don't have enough time for the same.”
If you haven’t come across it yet, find it here: How does a Data Product Strategy Impact the Day-to-Days of Your CMO, CDO, or CFO
This is Part 2, which addresses the following feedback:
We’ll arm you with a compact illustration that highlights the features and end products of every stage and also share a brief textual overview of all four stages along with references to involved stakeholders.
Overall, this article covers the stages of the data product lifecycle along with the purpose and end products of each stage, highlighting the data product's process of coming live through its ideation to activation that aims to improve the data-driven culture through better insights into business intelligence and data analysis.
Let’s look at the lifecycle at a glance first and then dive into the details.
Summary of every stage and involved stakeholders:
The design stage is the heart of the product approach. This is where thorough user analysis is conducted to track the most critical user problems and design solutions accordingly. The Design Stage answers questions such as:
The outcome of Design Stage is a collection of PRDs (product requirements doc) or data contracts, one for each output port. They outline the shape of data, semantics around it, quality conditions that must be fulfilled, and the necessary policy constraints applicable to that data.
It follows that the design stage requires the active involvement of Domain Experts who understand the business and can lay down a strategic plan for the data that serves their domain’s customers. The Data Product Manager becomes the conductor, strategically integrating inputs and launching the design plan on a collaborative interface to record inputs from different stakeholders.
Note that the title ‘Data Product Manager’ is quite novel, and it has been observed how folks with different or more traditional titles deliver the same jobs. For example, a product manager who owns an AI add-on or a senior data analyst overseeing a particular analytics solution.
Once the design is validated and locked for the first cycle, the same interface is accessible by data developers to activate the design with underground mappings.
The contracts formulate a robust view of the product or a semantic data model that enlists all entities involved, such as customers and orders, relationships between them, and most importantly, metrics.
This is where the nuts and bolts behind the data product are put together. Given data products cut across infrastructure, code, data, and metadata, ready-to-use resources such as compute, policy, contract, storage, workflow, and service become fundamental to a seamless development experience.
The design and develop stages do not necessarily need to be consecutive while you build a data product. A new change request can be added to the collaborative interface based on customer requirements, and the mapping in the development would change accordingly and rather quickly due to declarative development.
💡 Note the likeness to agile, where you don’t need to wait for different steps to end before pushing changes. It supports simultaneous as well as consecutive.
The end state of Develop is a Bundle. Data Products are developed as a package or Bundle, which is a collection of all resources and assets that help the data product run smoothly. The raw materials that go into a bundle include:
The aim is to release frequently and release strong. This entails continuous integration and continuous deployment (CI/CD), where each merge goes through cycles of automated tests and releases to maintain an optimum state of data product experience. The purpose is also to push better experiences or states in a non-disruptive way that doesn’t impact the end user negatively.
Executing this stage results in an activated data product, which is instantly populated in the data catalog and starts serving all applications plugged into the output ports. These applications may vary from a query interface, a CRM, an AI app, or even a dashboard.
The true power of this stage is, in fact, in the Data Platform that enables it. Optimal change management and release are the first and only priorities of the deployment stage, which makes this phase the critical pivot of the data product lifecycle. Every other stage fosters static assets, and this stage is where the changes are activated and pushed into the data product.
The significance of the evolve stage is that it is the change inducer for optimising resources, SLOs, and integrations based on both machine and consumer feedback. Right after a data product is live after deployment, all data around its operations and consumption patterns are recorded and analysed to send back meaningful insights.
💡 Like Agile, feedback can stream into any stage in the lifecycle instead of just the initial design stage.
This phase in the lifecycle also furnishes maintenance toolkits for developers to react efficiently to monitoring incidents and enhance key product fitness metrics such as time to recovery and uptime. The evolve stage completes the feedback loop. Having a sense of key metrics and objectives across different functions also helps resolve conflicts early on without disrupting ongoing builds and deployments and restarting from scratch.
This stage sends back Feedback or meaningful insights on metrics, applications, code, and infrastructure to influence the lifecycle phases and the domain experts in the next iteration of the lifecycle. This demands continuous monitoring and stakeholder involvement for meaningful analysis of observed metadata.
The feedback can be used to empower four top objectives:
It’s important to understand Data Products is more an approach than technology, much like Agile for Software. Agile went through its share of pushback from sticky processes and emerged as a must-have. Post-adoption, Agile became a game changer for software teams and products. Unlike Agile for software, which focuses primarily on code and functionality, the Data Product approach inherently fosters data democratization.
But an approach for software cannot necessarily be duplicated for data. We have seen that over most part of the last decade. Interestingly, the Data Product approach has changed that. It works for the core indefinite element of any data stack that is absent in the software stack: Data. And it needs to be actively enforced through a Data Product Platform that integrates with existing data stacks.
More about it soon!
