Wayfair’s Multi-year Data Mesh Journey | Nachiket Mehta and Piyush Tiwari | Shift Left Data Conference 2025

Talk Description

Wayfair’s multi-year Data Mesh journey involved shifting from a monolithic, centralized data model to a decentralized, domain-driven architecture built on microservices. By embracing Data Mesh principles, Wayfair empowered domain teams to take end-to-end ownership of their data.

Key enablers included a data contract management platform ensure trusted, discoverable data products, and the development of Taxon, an internal ontology and knowledge graph that unified semantics across domains while supporting the company's tech modernization.

Organizationally, Wayfair introduced an Embedded Data Engineering model – embedding data engineers within domain teams – to instill a “Data-as-a-Product” mindset among data producers. This sociotechnical shift ensured that those who create data also own its quality, documentation, and evolution, rather than relying on a centralized BI team. As a result, Wayfair’s data producers are now accountable for well-defined, high-quality data products, and data consumers can more easily discover and trust data through the unified catalog and ontology.

The presentation will highlight how Wayfair has adopted the “shift left” (pushing data ownership and quality to the source teams) and next heading towards “shift right” (focusing on consumer-driven data products and outcomes) to unlock business outcomes. This session will share both technical strategies and business results from Wayfair’s Data Mesh journey.

Additional Shift Left Data Conference Talks

Shifting Left with Data DevOps (recording link)

• Chad Sanderson - Co-Founder & CEO - Gable.ai

Shifting From Reactive to Proactive at Glassdoor (recording link)

• Zakariah Siyaji - Engineering Manager - Glassdoor

Data Contracts in the Real World, the Adevinta Spain Implementation (recording link)

• Sergio Couto Catoira - Senior Data Engineer - Adevinta Spain

Panel: State of the Data And AI Market (recording link)

• Apoorva Pandhi - Managing Director - Zetta Venture Partners
• Matt Turck - Managing Director - FirstMark
• Chris Riccomini - General Partner - Materialized View Capital
• Chad Sanderson (Moderator)

Wayfair’s Multi-year Data Mesh Journey (recording link)

• Nachiket Mehta - Former Head of Data and Analytics Eng - Wayfair
• Piyush Tiwari - Senior Manager of Engineering - Wayfair

Automating Data Quality via Shift Left for Real-Time Web Data Feeds at Industrial Scale (recording link)

• Sarah McKenna - CEO - Sequentum

Panel: Shift Left Across the Data Lifecycle—Data Contracts, Transformations, Observability, and Catalogs (recording link)

• Barr Moses - Co-Founder & CEO - Monte Carlo
• Tristan Handy - CEO & Founder - dbt Labs
• Prukalpa Sankar - Co-Founder & CEO - Atlan
• Chad Sanderson (Moderator)

Shift Left with Apache Iceberg Data Products to Power AI (recording link)

• Andrew Madson - Founder - Insights x Design

The Rise of the Data-Conscious Software Engineer: Bridging the Data-Software Gap (recording link)

• Mark Freeman - Tech Lead - Gable.ai

Building a Scalable Data Foundation in Health Tech (recording link)

• Anna Swigart - Director, Data Engineering - Helix

Shifting Left in Banking: Enhancing Machine Learning Models through Proactive Data Quality (recording link)

• Abhi Ghosh - Head of Data Observability - Capital One

Panel: How AI Is Shifting Data Infrastructure Left (recording link)

• Joe Reis - Founder - Nerd Herd Education (Co-author of Fundamental of Data Engineering)
• Vin Vashishta - CEO - V Squared AI (Author of From Data to Profit)
• Carly Taylor - Field CTO, Gaming - Databricks
• Chad Sanderson (Moderator)

Transcript

*Note: Video transcribed via AI voice to text; There may be inconsistencies.

" Right now we've got our next talk. Na, I see you up and on the stage. How you doing, man? Doing great, man. I wish I had your energy. Well, sweet. I know you've been here with us for a while. Hopefully you're enjoying it. I'm gonna let you share your screen and get rocking and rolling and we will.

Have your, your talk coming in. We've got another person coming into this too, right? Uh, yeah, there we go. Hey, I, I thought there was two people on this talk, I just wasn't quite sure. Yeah, let's start sharing screen uch. We'll get started. Awesome. Well, he's doing that. I start introductions. Um, so I'm ma Hello everyone.

So excited to be here. I'm very passionate about this topic. Uh, it's pleasure to be part of this community, uh, huge with me. Uh, it's from Bayfair. Uh, I, by the way, I left Bayfair a couple of weeks ago. Uh, so I'm a former head of data and analytics, uh, for global operations and enterprise now, but peers, you wanna introduce your yourselves?

Sure thing. Is my screen visible? Yes. Okay, sounds good. So, hello everyone. Um, I am Ush. I manage, uh, I'm a senior manager at Wayfair. I lead the data and AI enablement platforms at Wayfair. And, uh, my team composition is software engineers, data engineers, and AI engineers working towards building the next gen data and AI platform at Wayfair.

And happy to share about our data mesh journey today. Uh, awesome. Um, so for today's agenda, we're gonna cover just for those who don't know about Wayfair, just a quick overview of Wayfair and then we'll jump right into Wayfair's data mesh journey that started a few years ago. Um, and as most of you may know, it's not just a technical, it's a social technical paradigm shift.

So we are gonna talk about both the technical as well as social aspect of it, and then what's going on next, right? And so we'll leave some time for the q and a brand, uh, but with that I'll have huge talk about the Wayfair.

So Wayfair, we are one of the world largest online retailer for home. Uh, so if you are looking to decorate your home, give a, give a look to Wayfair. Uh, so Wayfair is enabling our customers to realize their dream home journey by the big catalog million of products in our catalog, uh, and with our brand of, uh, uh, with our different brand sites like all modern Birch Lane and our fiscal re stores.

So we opened our, one of the Wayfair Marques stores in Chicago, and the next one is coming up in Atlanta. So if you are around Chicago area, give it a look. It's an awesome experience. Moving on to Wayfair. Wayfair is not just an e-commerce company, but it is, uh, the company, like the company has technology at heart.

Uh, we have a wide selection of product and customized user experience. So everything we do at Wayfair is driven by data. So our data platform is part of the whole customer journey, starting from the search and personalization to our sales and service. Uh, and with the Gene ai, uh, picking up, we have been actively, uh, influencing some of the products which we are working on with Gene.

Ai. Muse is one of the products where we, our customers can, uh, curate their home using Genai. If you have not taken a look at it, please take a look at it. It's fun. Using Muse, the tech transformation journey, which Wayfair has been the data tool platform, has been really helping with the, with the, uh, speeding of the tech transformation using Gene.

I, so moving forward, like I will hand it over to NN has been the flag bearer of data mesh at Wayfair and has been with us, uh, with us on this journey of last four, five years of establishing the data mesh platform, uh, or establishing the data as a product ownership and impacting the whole cultural change at Wayfair.

So, thank you B. Um, so yeah, for those who don't know about Wayfair's history, uh, there's an operational side and there is an analytical side. Uh, operationally, Wayfair was one of the largest SQL Server deployment in the world, and, uh, it came at its own cost. Uh, there were a lot of replication happening when we needed data for analytics.

At the time when company was growing really fast data was democratized in a way where everyone had access to most of the data except very sensitive data. Uh, but that created a lot of strong, right? Um, and then on in 2000, early 2019, we created a separate Google Cloud, um, enterprise data environment called Eden, uh, to address some of this like replication issues.

Uh, but before we knew in, in kind of couple of years, it grew significantly as well. So we, while we address certain issue to not have run analytics on, on kind of operation data or applications of that, we had a different problem, which cost us like a lot of money. Uh, and that also it had its own accountability and trust issues, right?

So as you see on the, the, the numbers here, they're like drastic. It's petabytes of data, uh, but very few ownership and, and visibility issues. So if you go to the next slide, uh. Starts about some of the pain points that both producers and consumers had, uh, from consumer's perspective when they were using data to drive some key business decisions and insights.

Uh, and then they had issues. Uh, we did not know him to reach out to all the time. They were reliant on some central data teams. Uh, they in, in turn were reliant on some other data producers they in felt had other kind of service connections built in. So that was little by while that, uh, anyway, uh, meantime to remediate was really more than four days for when we identify issues like a storefront, changes that made missing click attribution, or there are instances where there are supply chain forecasting errors were not as accurate and then it's not yielding into more accurate actuals.

Um, so there are a lot of issues, uh, and then seeing like data producers had their own problems where there are thousands of consumers consuming that data, they did not know whom to reach out to when we were trying to make this like systematic and digital transformation related changes, uh, and the context, the level of context, right?

There are instances where we did not have like very common enterprise level data model at home. Uh, so everyone was using their own kind of for their own application, uh, their own dedicated data models. There are instances where, uh, a, a box that is traveling from inbound all the way to the customers and returns had like 12 different identifiers across the system, right?

So no common kind of context and, and ontology were enforced. Uh, to go to the next slide, we'll talk about your journey and how it all started, right? So for those who don't know about data marsh, it's really social, technical, uh, approach. We started, uh, along our journey of decentralization and, and moving from monolith to microservices.

Uh, strategic part is very important and so is the social and and technology aspect. So when we started the journey, uh, back in 2018 19 to really move into this more, uh, kind of decentralized domain-driven architecture approach, that's where we found we set a good foundation at all of the technology org level.

And then, uh, by the way, we don't have, uh, a central data office that manages all the functions, right? So our high level data teams, we are federated within the, the high level domains. Uh, so our, a group of us came together to identify how we can address this issue. Uh, since I was leading the supply chain domain at that time, we started with supply chain domain as early adopters, work with the producers and consumers of that domain, a lot of applications, uh, to identify what problem we are trying to solve, and let's associate that to a co to a business value that, that we're facing some, uh, impact.

Uh, last two years, we made a significant progress. That's where a lot of social, uh, top down, uh, bottom up evangelizing happened. There were instances where we were a lot of town halls and tech talks, and wrote a lot of white papers, work alongside, um, software engineers to train, then build this data as a product mindset.

Um, so that helped us establishing this whole data contract, that shift left approach, uh, uh, associated that with the data catalog, uh, piece, and then got a wider adoption across enterprise. So that included all the domains. Uh, and then obviously now with the script foundation built, uh, we really like the benefits, uh, quickly on any initiatives, any initiatives that are kind of tagged to a data product, a consumer data product.

Uh, and then you can have a full end to end that goes back all the way to who owns the data. So just to try to address those ownership and visibility issues, things are in motion. And then we are building and extending the rigor, uh, of our software that adopts, uh, process to what we call internally dial framework to be applied around the data products as well.

Uh, so that's just very highlighting journey. Uh, we'll definitely talk more about the technical aspects for that, I will hand it over to p to talk about tech. Thanks. So, uh, when we, uh, think about the data mesh platform, we think of it as a, a center, uh, place where we bring more data and establish a communication method between the producers and the consumers.

The, there are different aspects. What we took, uh, uh, took into building the data platform. The first thing, what I want to talk about is data contract. Uh, so data contract is one thing which helped us solve the problem of the lack of context, ownership and visibility. All three, what Nique pointed out. So while we are building the data contract, one thing I want to call out is, uh, why will somebody come onto the platform being a producer?

And there are so many different domains working on their different timelines, different applications. So the things which brought them onto the platform was, uh, the ease of creating the data contract, how cell service is, and then, um, also the ease of building the data quality rules. So, uh, how while building the contract, while building the standards, you're getting the benefit of, uh, better applications, uh, and better alerting and monitoring on top of that.

So that is one thing which, uh, ate mentioned the bottom up approach. So how do you make it easy for the, uh, engineers, application engineers building, uh, building the tool to come onto the platform and then also make it seamless, integrate with their application programming so that they don't have to take a different step to go to create a contract or establish it as a product mindset.

One thing which we slowly realized while building the platform was, uh, how the overall platform evolved into a data marketplace. So, uh, we started hearing from feedback from the consumer side of the word that, hey, it is easy to find a data, uh, easy to discover the data assets which are part of the platform.

So slowly we started growing this, uh, the platform, which started from one domain to more of a generic data marketplace, where now folks are, uh, trying to discover their data, discover the data assets before going and reinventing the wheel. So that is another thing we saw on the platform. Uh, the third thing was how easily you can control the access of the data.

So it is more towards, like everyone, every department, every domain is at different stage of their journey or their application programming. So we integrated the data access onto the platform. So if you're building an application and you're creating a contract, you can, at the same time, you can control your data access, uh, through the platform.

One thing we recently saw, part of being on, like when the gene ai boom was coming and Wayfair was moving towards innovation with gene ai, lot of our, um, gene AI data application data bot or data agents, uh, really got benefit from having one centralized context window or a semantic layer for their agents or bot to build the context from.

So for example, like first they get the quality of data to train on or to, uh, work on, and then second, the context building through the schemas what we have or the, uh, catalog of what we have around the data assets really helped us, uh, the data part of the gene AI or data bots to, uh, build that context fast.

So moving on to the architecture, uh, there are three pieces to the architecture. How we look at. One is the self serve nature of creating the contract. So how is how we can remove the friction of, uh, creating the contract? So the, our approach has been meet where the application teams are or data producers are on the D like they can create a contract through the CLI, they can create the contract through a Terraform module.

Uh, they can be a proactive about validating the contract through the plugin, or they can be reactive through our, uh, schema validation. So once the contract is onboarded, uh, along with the contract, you have the ability to create the quality rules as well. So we use DataFlex for our quality rules. So once the, uh, producers create the contracts, enable the quality rules, there is a data contract service which coordinates the schema validation or the contract validation and the quality validation and reports it to data hub.

Data hub is our marketplace. Where you can get the current state of a data asset. What are the quality rules enabled? What are the contracts enabled? How many times it has violated the contract? And at the same time, we also have the notification built in, uh, based on how an application or data producers want to do it.

Do they want it with Slack, they do. Is the data asset a critical asset where they want to set it as high tier and have PagerDuty integration? One thing which I specifically want to point out is Taxon, which is our Ontologic platform. So we were running into issues while establishing, uh, standards on our data model.

So that is where we did build a homegrown ontology platform, which is taxon, uh, which enables the domain data model managed by our domain data and also helps us build the knowledge graph, which has also helped us building the context of our journey application. Uh, we are running short on time, so I will skip and take the questions in the last, uh, just one last call out.

It takes a village to build the platform and the maturity, what we have achieved on the platform, uh, thanks to my team and also, uh, uh, we have been working with, uh, data hub and, uh, Gable throughout the journey to understand the standard contribute back and the same time mature the whole, uh, shift left con, um, concept.

I will hand it over to, to talk about some the organizational transformation and then wrap up. Uh, yeah. So organizationally, this is like a few changes we made that is kind of in innovating and it worked out really well. That may help you all is one thing is, uh, instead of having the center data teams to drive this both on the producer and consumer side, I called out a, a, a small team called Embedded Data Engineers and they came up with like this whole checklist that manifesto to, and make sure that there is truly driven data ownership owned by the source data producers.

Uh, and one of those kind of checklist was make sure making sure that data contract is established. They have their own project, a dedicated project where they own maintain their events or APIs or even like a flat tables, uh, that are treated as a product and all of them should have data contracts. So that worked out really well directed as a consultants, uh, working on a one domain at a time, and then slowly green to more automation to cover more domains.

Uh, so that really drove this data as a product mindset on product ENG and engineering teams, uh, including product managers who really consider data and not as a byproduct, but really like their own, uh, data product. Then data studios, the taxon, like Paige mentioned, played a key role where we associated or we created a, a common place to register your domain models and then, uh, ensure that we are incorporated into chema chip parts of the, the data contract that then, uh, it also helped us in building more context around LLMs.

So when people are like consumers are asking more plain, simple English questions, we could leverage those contexts to translate from text to SQL and give them the results much faster insights instead of going through the traditional way of doing it. And then service mentality and just making sure the business impact is, is driving all the priorities.

So what's next? Uh, I just wanna call out that I think our shift left approach is well along the way, adopted by the organization. Uh, but shift right is where we are kind of also focusing on. It's not one or the other. Both are equally important, but just focusing on the consumer data products, packaging that consumer data product and showing the full lineage going all the way where data is produced and vice versa for data producers to understand who their consumers are because now they're aware of their consumers and the business impact and the use cases.

Uh, so that's where what's happening. And then the next slide, just have quick takeaways for you. Our main thing to call out, do not ignore the cultural change and transformation, right? It's no matter how much you invest in technology, that is very important. The adoption is very important. Like Chad was talking about it earlier, uh, being wrote a lot of white papers.

We actually combine best data principles and product principles and combine them and evangelize that concept to make, make the users where they are, right? If it's a data producer, they have their own problems and visibility. So we explain that aspect. And consumers, we kind of consider their pain points and kind of evangelize that aspect.

Uh, tech platform side of things. Uh, investments we made really help us in tech transformation aspect. The domain model store and things like that. And where we are going, like how can we scale generating the data contracts automatically using gen AI and things like that. Uh, governance, the taxon, we already talked about it.

It provides a good federated and a domain optim balance. Uh, and at the end of the day, like, do not treat this as like a tech sponsored initiative, uh, aligned with the leaders, aligned with the top business priorities, have the business focus and problems at the, the, the center, and then connect all this kind of solutions at, uh, at the end of the day.

And this is not a process. Uh, so it's a lot of more work to do as more and more problems would be identified as we move from both DevOps to AIOps. Uh, but that's what I just want to leave you all with and happy to answer any questions.

Woo hoo. All right, so we already got some questions trickling through and I really appreciate you all for giving this awesome talk. First question from Pete in the chat is, was your team the embedded data architects or did the embedded data architects report to their respective departments? That's a good question.

So, uh, I think it's a little bit of both. So there are the, is more the, the sample like this, the team that I created that were more like the data architects who also understood the software world much better so they can really cross train. So instead of fishing for them, they can really teach them how to fish and then create that.

I think that previous talk we were talking about, whether it's a data engineer becoming a full step data engineer versus where they wait, but we went on that route to really make a full stack engineer really aligned with more data architecture and all. And then many times the engineers said, our goal is not just to target the architect, like senior level profiles, but how like on one and two, those engineers also treat data as a product, right?

So that's where the, the Anology piece came in where the senior folks focus on the data stewardship and defining the data model in one place, and then they can leverage, they don't need to have that architecture knowledge, but they can leverage, rely on that tool through their like CLL that they are working with other, the core gen and then they can be empowered that way.

So it's not limited to the top senior folks. Everyone is kind of empowered to do that. Right. On uh, anything to add to that, since we got two folks on here, or should I hit the next question? Well, I, I think, uh, tric Kate covered it, right? It is just not one way. It's mostly you move the data functions closer to the application teams and that's where embedding the data engineers helped.

Uh, rather than having a centralized data engineering team, the more you move the data practices closer to the application teams, uh, they build that, they build that thought process of owning the data as a product and learning the best data practices rather than making it as after taught. You can think of it as more of like, I'm writing an application, why not think about like how the data would be consumed?

So the whole shift left concept, embedding the data engineers enable start. Uh, all right. So I got another question coming through here from Paul. When you say it takes a village to build a data function as mature as Wayfair, what is the size of the team? What kind of village are we talking about here? I can start and probably Nagate can jump in.

So actually the team which started building the platform was very small and, uh, was like four engineers iterating, uh, building, uh, working as a Tiger team with all our domain partners. Like Nagate team was on a platform. We were four engineers building it. Uh, there were other engineers plugged from the domain teams so that we can get the real time feedback.

So team was small, uh, close to four, four engineers, and then a Tiger team had one engineer, uh, from each of the domains. So that's how we started building up the platform. What I mean by village is, uh, we were working with, with a lot of our partners. In this case we are using data hub and uh, we were like discussing, an educator was discussing with Chad on overall how the design strategy and influencing the design overall the platform.

So working with Google Data Hub and also some of the folks at Cable, that's what I meant by taking a village. The team was very small and very focused on executing on this, on the actual, on the ground. So it's not about the size of the team executing, but when I mean villages, the overall ecosystem, what we were building along with external partners and Google.

Excellent. Great. Yeah, and just to kind of share the, the, the scope of Wayfair, right? Wayfair was very, uh, engineer, heavy tech heavy organizations. So there are like about couple of thousand engineers and there are about thousands of consumers internally. Like we're really supporting suppliers or customers or internal stakeholders through the function.

So from data producer and data consumer side, there were a lot. Uh, and then the team that built it, like Perche, there is a smaller team, really group of very smart people. That's what Wayfair does best, right? Hire very best talent. Uh, and then they partnered with folks like we mentioned, uh, to really build this innovative platform and, and evangelize this and evangelizing took a lot of other leaders and collaborations to happen, right?

That's the where the social part comes in. So speaking of that, uh, evangelism, I think one thing that was a question that came up in the, some of the last talks is who spearheads this initiative and then who owns it? Yeah, it, it depends. I think anyone who is passionate about this topic, uh, is a good initiator, right?

Uh, and that's what happened in our case. Uh, and then you have to identify the people who are facing the same, who face, who face, who share the same passion and who are facing the similar challenges, right? Like in our case, it was really the central team that initiated this, the central data team that initiated, because many times they were in the center of both the producer and consumer.

No matter how much work you are doing, your data is nobody's problem. But then when something goes wrong, it's everybody's problem or it's, it's kind of blame it on the central team. So that's where like you done on a shirt. Exactly. Sorry to interrupt, but that was too good. Not to, data is nobody's problem until it's everyone's problem.

Yeah. And that's, that's, that's reality. And not just with Wayfair. I know many large organizations face the same thing, right? Like consumers are never satisfied and say, no, I have perfect data to drive my business value. Thank you so much. It's a thankless job. We all know that. And same way producers, they don't care about data, right?

They care about like what applications and business value they are driving directly and just bringing them closer was the  📍 main driver. And then talking in the language of both these personas, like really helped us evangelize this a lot, right? Where we were meeting them where they are, and then bringing the right thought leaders and, and uh, kind of voices, active voices in this that really helped, uh, drive this culture change.

All right guys, there are some really good questions still coming through in the chat. Um, I'm gonna ask one more and then we gotta jump to our next awesome talk with Sarah. I hear a lot of discussion around engineering teams, but when you have a strong business analyst community who want to self-serve and manage their own world, how are you, how are companies enabling this?

Or are they just rebranding analysts as engineers? That's a very good question. Uh, and I think we did not differentiate, like I know some people say like, producers are really software engineers and consumers are analyst community. First thing, what we did was, hey, depending on the use case, you could have a software engineer as a producer and another software engineer as a, as a consumer.

Or you could have an analyst who is producing some data, right? You are doing some gold record creation in medallion architecture. And then there is reverse ETL happening where that is integrated for ml AI kind of work. And then the analyst becomes a producer and, and then the others become consumers, right?

So no matter what your use cases are, what role you play, whenever you are exchanging the data and changing the form of the data asset, you need to have a data contract. And that kind of enables this daisy chain of handshake that everyone needs to follow. But second part of the question is what about the analysts that they are engineers and all?

Yes. I think that's where the world and technology is going, right? Like with AI enablement, like I think and intergeneration changing the analyst part can also be automated to a certain extent. To get the date, the insights or the business value out of data is much, it takes much less time than the traditional way of moving data from on multiple places.

Excellent. Well, we'll keep it moving, but there's a few q and a questions that are coming through, especially around what you would optimize in your six year journey that I would love to see. If you guys can stick around and type it in the chat, that would be incredible. For now though, I see Mark popped back up on screen."