Adopt A Product Mindset To Maximise Value From Microsoft Fabric

TLDR; We recommend adopting a product mindset and embracing the concept of data products to maximise the value from Microsoft Fabric. This will help to give your team purpose, build empathy with users and focus finite resources on high impact work.

In May 2023, Microsoft announced Microsoft Fabric. It extends the promise of Azure Synapse Analytics integration to all analytics workloads from the data engineer to the business knowledge worker. It brings together reporting, analytics, data science and data engineering on a new generation of lake house infrastructure. Delivered as a unified SaaS offering, it aims to reduce cost and time to value, while enabling new "citizen data science" capabilities. See Ed Freeman's Introduction To Microsoft Fabric for more background.

The objective of this five part series of blogs is to help technology leaders assess the strategic implications of Microsoft Fabric. So, if you are a Chief Technology Officer (CTO), Chief Data Officer (CDO), Director of Data and Analytics, Director of Artificial Intelligence, Head of Data Science or any similar senior data leader role, this article is for you!

Over the coming months, Microsoft Fabric will move from public preview to general availability at which point it will be feasible to start deploying production workloads on the platform. This blog will help you to identify some of the actions to take over the next 6 to 12 months to get your organisation ready to successfully adopt Microsoft Fabric and maximise the benefits from it.

Harness the power of a product driven approach

In many organisations data has historically been viewed as a "nuisance": a by-product of running the business. Data and analytics investments are often viewed as an overhead, as something that consumes the organisation's resources. It is also common for centralised data teams to operate as "order takers", with little connection to the wider organisation or sense of purpose in the work they are doing.

By thinking of "data as a product", there is an opportunity to shift into a new mode of operating where data product teams become innovation engines for the business, actively looking for opportunities, seeking out where value can be created and challenging the status quo. To succeed, these teams will need to develop a good understanding of the product lifecycle (see illustration below), define what a "data product" actually is, and what's required to discover, build and own a successful data product.

Build empathy with users

One valuable aspect of a product mindset is that it relies on building empathy. Successful data product teams place significant emphasis on collaborating with end users early in the lifecycle to build situational awareness. By taking this approach they develop a common language, so everyone understands what is being considered, and can articulate clear success criteria.

This enables the team to make an important shift from "order taker" to "innovation engine". This begins by working closely with stakeholders to understand their objectives, the tasks that they need to perform, and impediments that they face.

Focus on actionable insights

By building empathy with end users, and the deeper understanding that comes with it, data teams will become goal driven. In data and analytics terms, this is about capturing the goals of the users and wider organisation, then identifying how data can be used to deliver actionable insights that will enable them to achieve those goals.

At endjin, we apply this principle to all data projects through our Insight Discovery process. This ensures that the data products we build have direct value to the organisation.

Once discovered, these data driven actionable insights represent small atomic units of work that can then form a prioritised backlog for the team.

Fail fast

Teams that adopt a product mindset take an iterative approach to delivery, delivering data products that tackle one actionable insight at a time. In taking this approach, they are working according to agile principles and they are able (and willing!) to "fail fast" or to "pivot" based on what they learn during each iteration.

Data projects are prone to failure. Often this is because the data needed isn't available or of sufficient quality. In other cases, it could be because the model you have developed can't achieve the desired accuracy to put it into use.

Product minded teams are aware of the propensity for data projects to fail and seek to identify this situation as early as possible.

A high proportion of data projects fail. That's fine provided it happens in the early phases of the lifecycle.

Balance cost versus value

By adopting a product mindset, the data product teams start to "think like a business" adopting a proactive approach to delivering value while also accepting the responsibility for resource-allocation decisions by creating a clear link between data products, business value and total cost of ownership (TCO), balancing inputs and outputs accordingly.

This leads to solutions that are not "over engineered". It also allows mature conversations to be had with stakeholders around prioritising features to fit within a budget that is proportionate to the business value.

Catalogue and develop migration plans for existing data assets

Your organisation is likely to have existing data and analytics assets. We recommend cataloguing these assets and assessing them using our simple analysis toolkit to generate actionable insights:

As illustrated above, this toolkit will allow you make data driven decisions by plotting each data and analytics "asset" (where an asset is a significant component such as a report, machine learning model, pipeline or data repository) according to two axes: business value and technical efficacy. This will enable you to categorise each asset and take appropriate action as follows:

• Protect high value assets. Leave on existing platform for now;
• Invest in high business impact assets that have low technical efficacy. Seek to migrate to Fabric to lower the TCO, and address technical debt that could put the business value they generate at risk;
• Tolerate assets that are strong technically but are not having a significant business impact. Challenge relevant owners to find ways to generate more value from these assets;
• Retire assets that are not providing business value and are weak technically. Do this to release capacity in your team to discovery, build and own higher value assets.

This approach effectively retrofits a product mindset to existing data and analytics assets in the organisation. Enabling you to make informed decisions about the future of each asset based on the value it is delivering to users versus its TCO.

Establish an internal data product marketplace

Once you start to implement the data product lifecycle (see illustration above), a key capability to support the "Grow" stage of the lifecycle will be to provide a means for data products to be registered, documented, promoted and distributed for use using Microsoft Fabric.

There are a number of options available on Fabric to provide a "product marketplace" ranging from exposing chosen tables in lake house via a SQL end point, leveraging the "OneLake Data Hub", using datasets within the Power BI service or layering Purivew over Fabric. All options need to considered and a suitable solution chosen based on your specific requirements. We recommend experimenting with these different options by signing up for the Microsoft Fabric trial.

Final thoughts

Organisations should use the time available before Microsoft Fabric becomes generally available to explore how a product mindset could help their data teams to discover, build and own high value data products that will have a significant impact on the organisation. With Fabric there is as an opportunity to make a significant step forward in operating model, not just "doing the same thing but on a different technology stack". A product mindset can help drive that change.

There are many resources out there to help you develop a product mindset, this includes our series of blogs on new product development.