Fulfil is an ERP designed for DTC brands like Ridge, HexClad, and Grüns. Manage orders, inventory, accounting, purchasing, fulfillment, EDI and manufacturing in one place. Native integrations with Shopify, Amazon and 400+ 3PLs (no middleware required). Go live in 8–12 weeks with fixed-price implementation (no third-party consultants).
Get a Demo“I just don’t hear a lot of good reasons why this can’t be automated”. Can an algorithm allocate ad budget better than a human media buyer? Olivia Kory (CMO & Strategy Officer, Haus) joins Connor MacDonald (CMO, Ridge) and Connor Rolain (Head of Growth, HexClad) to unpack incrementality testing, marketing mix modeling, and early agentic media buying. Geo lift holdouts and attribution debiasing point toward a future of automated budget decisions. Olivia shows how A/B testing replaced ROAS benchmarks at Haus. Control groups and treatment cells now anchor every incrementality read. Connor Rolain also shares how one conversion campaign test reshaped HexClad’s Meta strategy. The team ties it all back to multi-touch attribution for faster decisions. Nominate someone for Haus' Top Marketers of 2026: https://haus.io/top-marketers Powered By Motion https://9ops.co/motion-runneth NeonPixel https://9ops.co/neonpixel Richpanel https://9ops.co/richpanel Proppel https://9ops.co/weareproppel Aftersell https://9ops.co/aftersell-mops Haus https://www.haus.io/operators More from Operators Operators Portal https://portal.9operators.com/dashboard Operators Newsletter Chapters 00:00:00 Two CMOs, No Cody 00:03:50 Top Marketers Awards 00:11:27 Incrementality Grows Up 00:14:34 Beyond Raw ROAS 00:18:06 Optimizing Meta First 00:23:27 Brands Need A Baseline 00:27:29 Progress Over Perfection 00:32:07 Is Your Holdout Real 00:38:41 Long Holdout Tests 00:44:16 Should Robots Buy Ads 00:57:45 Solving The Cold Start 01:04:25 Overrated Or Underrated
Sign in to save notes on this episode.
All right, we are back with another episode of Marketing Operators. It's a special one. We've got one of my favorite people, great guest today, Olivia Corey, Chief Marketing and Strategy Officer from our favorite Geolift measurement toolhouse. Olivia, how are you?
I am doing great. I can't believe Cody is not here, uh, but I don't know, maybe I'm, I'm a little bit sick of Cody at this point, so this is for the best. I'm excited to, to mix it up with you two.
Totally. Yeah, I, I was bummed. Cody's sick. He's been burning it at both ends. You know, I heard he's been going to Lowe's. He's been doing chores around the house. He's been spending a lot of time on X tweeting about AI. Unemployment, I guess, is like really weighing on him to the point where he's missing podcast episodes. But I think we're gonna be able to kind of scrape it together here without him.
Are you guys worried about how much he's taking on? Like, I just, I think it's, and I'm wondering if you ever think about this, uh, whenever my house chapter ends, I'm gonna like lay on a beach for for at least a month, uh, and do nothing. And he is, he has like, you know, 47 consulting calls. He's doing landing page services. He's starting new brands. Like, I, I get it, but like no break in between.
I would be on your team and I would go much longer break.
I like visualize it during yoga. I like visualize the beach that I'm gonna be laying on for at least one month.
I think he's planting a lot of seeds though. You know, he's like, he's like, what do I wanna do? He's like, do I wanna consult? Do I wanna make landing pages? Do I wanna start brands? He's like, you know, he's dipping his finger in a bunch of different things. But I agree, I'd be, I'd be hanging for a little bit on the beach waiting for that idea to strike. And then I'd be that once that idea hits, it's all in.
One funny thing you, you guys will appreciate is, um, do, do you know, did you know that Taylor Holiday hates voice notes? Like with a burning passion.
I didn't know that.
Okay, so Cody and I are just yapping, voice noting Taylor just to, to bother him. So that's what Cody's been doing in his unemployment is yapping at Taylor.
Yeah, yeah, yeah.
No, I know Cody's got a bunch of cool stuff going on. Um, I like the idea. I like teasing him about him just rela— like him just being, spending a bunch of time on X talking with Claude, using Fable. Um, he told me he was going to Lowe's more. what he told me the other day. So I'm like, he's got a little bit of time. He's doing more things than he, he, uh, typically does. But, um, you know, I wanna say one other thing. Olivia, you recently became CMO of Houzz, and I'm, I'm just happy to have another CMO on the podcast finally. You know, we've got Connor as head of growth, we've got Cody as CEO position. Like, I haven't had like a CMO to really, you know, talk about the role with in a while.
Connor Rowland, your friend Cam just became a CMO, so we've been, We've been DMing on how we'll be sharing war stories. But yes, I wanna, maybe for a future episode, I would love for you guys to put yourself in my shoes and say like, what would you do as CMO of Houzz? Like one of the big open questions is like our paid, should we be doing paid? Should we be getting into the Meta ads game? We haven't done much here, but I mean, we have a lot of creator content, including you guys. pulling clips from the marketing operators and boosting them.
There is a path to that. I think this is a good segue into what I wanted to ask about next, because it feels like the best, like, B2B SaaS marketers right now are doing things like great customer advisory boards, great, um, you've got the, what's the Hamptons day trip with Shoplift and a couple other ones are like doing a helicopter into like a Hamptons house. Like it's getting, there's more dinners, there's things like that. It's like, it feels like B2B marketing has gotten really, really fun recently. You guys are rolling out a top marketers sort of accolade that you're gonna be giving out, which I think is actually like the best form of marketing you guys could be doing. Um, could you talk a bit about that?
Yes. So I am so pumped about this. We are launching the Top Marketers of 2026 program. I'm spending a lot of time here. Uh, think Forbes 30 Under 30, but more objective and you don't end up in jail.
Is that?
No, no, no, no. To be in all seriousness, like, I think it's no surprise kind of based on what we do and like kind of like the level of sophistication these brands are at when they adopt incrementality. I think it's like no surprise that we work with some of the best marketers in the world. Like, we are constantly told that the community around Houzz is best in class. It's part of the reason people love working with us. And so many of these customers are like not on the podcast circuit, you know, they can't be for whatever reason. There are restrictions. And so, we wanna celebrate, the idea here is like, what if we celebrate and spotlight the marketers who are driving real incremental outcomes in their business? And so, trying to make this like more objective and not just a standard industry list where it's about who you know. And so, really kind of celebrating marketers who are doing cool stuff. I think we'll have some like featured categories around AI, creative testing velocity, but we'll, I mean, the nomination, Criteria will be pretty lightweight. You're not gonna have to like fill out, you know, a whole application. But what's really cool is my judge who I'm bringing in to help me is Connor Dalt. He is my top marketer of 2026. And I told him I'm gonna embarrass him and he was so mad at me, but he, because he's gonna be a judge with me, he can't be included on this list. But I just wanna shout out to Connor because I have learned so much from him. Like the, the thing he is just so uniquely skilled at is he is able to kind of weave between the big creative idea, the big swing, and the really heady performance marketing finance world just in and out so seamlessly, um, that I have just been so impressed. And like, obviously Groon's had an amazing outcome. He'll tell you he didn't have much to do with it, but, but I disagree. Um, so he's gonna help me judge here and This one's fun. We have our customer advisory board meeting in person every February. It's looking like it's gonna be at Pebble Beach in February of 2027, and a subset of the winners of this top marketers program are gonna come to our CAB event and hang out with, with the customer advisory board. And there'll be stuff for non-golfers too. My founder's a big golfer, and so I told him I need, I need to do this for him, but There'll be a, a non-golf track. And so, I'm just, I'm pumped. We're, we're, by the time this episode publishes, there will be a landing page. We're gonna nominate, of course, some of our customers, but this is also open to the public. Anybody can nominate, like you can apply, you can nominate yourself, you can nominate someone on your team. And we're excited to, to see what we get and just celebrate some of the amazing work that's been happening.
Awesome. I think it sounds like a super fun program. I've got 2 points here. One, I would love for there to be a, not saying I'll be on the list, but for the lucky 26 who make the top 26, uh, physical trophies. I think we need way more physical trophies in our lives. Like we all got them growing up playing, you know, AYSO soccer or whatever. Um, I would love anybody to possess a 2026 Marketer of the Year, like Oscar equivalent. So that's one. Do you think we could get that done?
Connor Dalt, he, when we were jamming on this idea, he said there's a reason paper plate awards have stood the test of time. And so his idea was a paper plate award, but maybe we can upgrade that to a physical trophy.
Okay.
Yeah, I'll talk with him too. I'll, I'll give him a pitch. Um, okay. And then just throwing out a couple other names. Uh, Connor Dalt is a good one. I guess he's technically disqualified. Connor Rolling, pretty good.
Connor Martin, throw my name in the hat. Come on.
I've got Connor Ataya. Founder of Kantara. I've got Conor Gallagher. I've got Conor Sunderland. I've got Conor Daye, and I've got Conor Gross. All some of my, just really, I've got no bias here. I've got no, no agenda. I just think they're really smart marketers. Just some of your favorite marketers.
Should be considered.
Yeah. Some of the casual.
The CFO of Groove is also a Conor. I don't know if you knew Conor Stastny.
I, I just recently met him. Yeah. They had 2 Connors. That explains the great outcome.
That is wild. We could have a Conor track. At our CAB event.
Yeah. Yeah. Yeah.
Who do you think should be on this list? Like, who do you look up to in this space? And, and I think one thing that, that tends to happen is you conflate the person with the brand they're at. And I think Connor Dalton is a good example of like, again, I, I think, um, I, I think both there, like the, both the team and the product created that outcome. But who do you guys admire as marketers in this space? Not just like the brands you admire, but the, the people you're learning from.
I think Matty Martin from Cadence right now is just on an absolute tear. He's one of those guys that's, I think, like transitioning from doing it all himself to like slowly hiring like leads for his team. But like, they've been on an absolute tear right now. And he's been the one that has really been like, not only driving that strategy, but I think also pushing a lot of the buttons, which is pretty impressive over the last couple of years. Now he's finally like, like he, he WhatsApps me the other day. He's like, hey, we just hired a creative strategist. Hey, we just hired a DR, like video editor. So like, I've been very impressed with With him and what they're doing in spite of not having like a super large team yet.
He's great. He was on the pod. Great episode. Um, I go, uh, I'm a big fan of Nectar Mattress. I've talked about that for a while. Scott McLeod was technically chief of staff for a while, extremely strong marketer, way more under the radar. And then we've had Jared Brody on the show who's a VP over there. They're great. I also love, um, back to the B2B point, which becomes a little bit more meta, but, um, Rabah, who I've never like spoken to. But like crushed it at Vermont, crushed it at Triple Whale. He seems to have been highly influential in the way that some of these like e-com SaaS vendors are like deciding to bring things to market. Um, so I'd, I'd, I'd consider him.
Keegan Tiggs from David, VP of marketing, ex-McKinsey, just coming in there, figuring it out on the fly. That's, you gotta throw his name in the hat. I mean, they've had a, an epic 2-year run.
Keegan's the man. We were, we were in Aspen with him. Uh, really good. Okay, this is cool. Well, when, uh, when we, when we launch this, you can formally submit. We'll review the applications and, uh, and we're gonna do a, like a special kind of open house episode, uh, to reveal the, the winners.
Motion just dropped their 2026 Creative Benchmarks report and it's been getting shared everywhere. Slack channels, LinkedIn, Twitter, sharing it in our private group chats. And it's great because everybody's been asking the same 4 questions forever. What is normal? How many ads should we actually be shipping? What is a healthy hit rate? And which formats really win? The report analyzes over 575,000 creatives from 6,000 advertisers and over $1 billion in ad spend to answer these exact questions. And the report has some really interesting findings, like the fact that only 4 to 8% of ads actually become winners and over half of ads actually lose. And for Motion customers, this report is especially helpful. You can upload it into your Motion dashboard with their Runneth AI chat and compare it directly against your vertical benchmarks. Hit the link in the show notes. I promise you won't regret it. And as always, go to motionapp.com and tell the marketing operator sent you. It's funny having you on, Olivia, because we talk about almost every episode is an incrementality episode, but this one's a special one.
Um, I finally get to be here and, and participate in the conversation.
Yeah.
Yeah, 100%. Okay, so I wanna talk about like the way people are using incrementality today. I've kind of got like one of the shifts that we've had in strategy over the last couple years, but maybe just you could start us off with, you see more than anybody else a number of brands, not only like implementing incrementality and measurement strategies today, but also how that's shifted over the last couple years. So maybe you could give us just like a brief state of the market. One of the questions I had was if you've seen any interesting changes in how people are implementing these strategies. So, some sort of like wide scope as far as like, where does the industry sit right now? As well as like, if there's any interesting trends that you're seeing over the last few months.
I talk about our Houzz journey in 2 chapters. Like the first journey, or the first chapter of Houzz was like convincing people that incrementality mattered. This was, you know, 2021, 2022. That term was very niche. Like it was reserved for the Airbnbs of the world, the Ubers of the world. And Zach and I just like weren't even sure about the TAM here of like, do enough marketers kind of like care about this to go evangelize it in their organization? And we're through that. I mean, that is like very, very clear that you can tell that, you know, everybody's running incrementality tests right now. And we've entered chapter 2, which is like, how do you create an operating system around incrementality testing such that you are using it to improve business outcomes. And we just published a case study about this with Gamebridge this week around like, how do you string a series of tests together to actually drive growth in your business? That's what we're all after, right? When you talk to a lot of brands, nobody's really interested about the idea of like just cutting a bunch of channels and getting more efficient. Although, you know, if your goal is, you know, like a PE exit, like, yes, like we've had a few of those. But for the most part, people want to see, they want to unlock growth. Like they want to drive growth with this stuff. And so that is what we've been working through with our customers is like, how do you actually use incrementality testing to unlock growth in your business? And that is, that is like why we have a services component to our business. It's hard. It's like really hard. What do you do when the number disagrees with something else you're seeing internally? Or You need to turn ads off in part of the country for some period of time to get a precise read. Like, it is certainly not for the faint of heart. And so, that's why we have this like kind of software plus services strategy to like make sure you're actually getting value and you're getting ROI, and you can point to these tests and say, this impacted my business in a positive way. And so, that's the big shift that I've seen at least is, You know, they're, they're thinking about it more as like a kind of culture of continuous experimentation than a, than a report card, if you will.
One of the big shifts that we've had over the last, it was this time last year where, cuz it, it comes from 2 things and I hear this all the time when speaking to people. One, we get a number of tests that I think it's really hard to, um, you know, perfectly measure incrementality with a holdout in terms of like getting a direct IROAS. We run, and I talk about this all the time, like relatively short tests. And for that reason alone, we're just gonna like miss some of the value that's being created. So like trying to like scorecard and like grade some of these channels on like a 3-week test basis is like not all that accurate. Um, so we end up getting like what look like subpar results when, if you, if you, you know, widen the aperture on our business, we're like spending more money, we're growing profitably year over year. Like, Like fundamentally everything's happening the way that we want it to. Um, but the way that we test often leads to like some relatively conservative readings on the impact of some of our channels. So that's one point that I'll make here. The second one is, um, when, when we would prioritize like, okay, let's just get what is the incremental ROAS of Meta, we would just be getting it for a snapshot of time. So not only is it just for this 3-week period, maybe you're getting a 1-week post-treatment window, but it's also just February 2025. And like, that's not super applicable moving forward. So where we've shifted, and I'd love your perspective, is treating incrementality testing more as A/B testing. And we have direct actions that we can take from that. And what I mean is to say, the cleanest example, and we have many of them now because it's almost exclusively how we're running our measurement program, is going into BFCM, The 3 weeks beforehand, we launch incrementality tests on our top 2 channels. And our goal is really to just get the read of which one is driving the best incremental dollars at its current scale. So when we need to spend more money, as we do going into BFCM, we have really up-to-date data as to where our dollars might best be spent. That doesn't tell me the IROAs of YouTube or the IROAs of Meta, but I feel very confident that we are now making a better data-driven decision around where is my next dollar best spent to drive incremental results? So, it's like a shift that we've had. And I think that's a, I think it kind of maps to what you're describing, where we started out and we went channel by channel and we measured, and now we're thinking about it in a more like progressive way. And, um, and I think of it almost like, yeah, directionally, we wanna be taking action from these things. Do you think that is more of a Ridge-specific thing, or are you seeing that across the board?
I love this, by the way. I'm, I'm really glad we're having this conversation because A lot of our customers listen to this podcast, and I think there will be kind of good, tangible advice for folks who are listening here. I heard you talk about this on a pod, and I was talking to the team. Do you think— I love it, by the way, the like A, B, I think that's very actionable. There is a very clear action after that test on this thing is better than this, let's roll it out as the winner. Do you think that you're there because you've run— I don't, like, you're one of our highest Velocity testers, like, do you think you're, you're at this stage because of the maturity of your program? Or like, if you could go back, would you have started here? Uh, like, do that? Because I, I know you started with channel baselines, like, give the audience maybe some advice here on like, do you think this is just a function of like, you've answered a lot of core kind of baseline questions and now you're moving into optimization? Or do you actually think this is maybe where some of these brands should start?
Yeah.
When we started, we went channel by channel and we measured the incremental ROAS of each of those for these snapshots of time. Um, and there was value in that. Like we found, you know, non-branded search wasn't as incremental as we thought it would be. And that's very, I think, particular to our business. Um, more people search for Ridge Wallet every month than they do for men's wallet. Like there's just not a lot of in-market demand. And I think we have to be very careful about our non-branded search not capturing existing intent. Um, so anyway, so we, so we go channel by channel. That's kind of how we roll out those tests. Uh, and I think that was important and we got a lot of value. And then, and then I guess what I'm describing now, our strategy shift does come from maturity. I guess what I would say that in hindsight, you know, I would love to just be able to show up and be like, hey, we need to drive a 2x incremental ROAS. Like, that's where we're profitable. That's where we're gonna hit our goals. Let me just go channel by channel and figure out whether we can do that. And I just think it's a little, in hindsight, I think it's a little bit more complex than that. So I do think the channel by channel is a good testing strategy and a great way to kind of get the ball rolling and how to leverage geo lift studies in how you want to approach measurement. But at the same time, like I have found it a little bit more nuanced or tricky to get like a true readout given the snapshot in time component given. The snapshot in time component in terms of like the time of year, again, February for us is way different than June. And then also the fact that we are measuring some of these channels on like relatively short timeframes when, you know, 40% of our customers every day say they've heard about us for over 6 months. So I'm like, it's just like a lot of, um, uh, nuances there that I thought we'd be able to better account for. And I think that's part of what drives the strategy towards this more like A/B testing directional approach.
Can I add a note on that? 'Cause I think a lot of brands like come in and they immediately start running holdout tests on their hero channel, like right, like straight up holdout, right? Like classic A/B, no spend, spend, and they're trying to measure the incrementality of Meta as a whole. I don't think that's a good first use of your tests. Like if you got to become a, to a size where you're onboarding with a house, you know, presumably you're probably like a healthy 8-figure brand. Presumably you got there on the back of Meta. I'm not saying you shouldn't do those tests. I think you should save those tests for a new channel expansion. But I think the move is to spend your first like 6 months with house actually optimizing meta with holdout tests using the directional approach that we're talking about. Not like a true channel holdout, but like, how do we unlock more incrementality out of meta? So like, let's start testing view content optimized stuff. Let's start testing like maybe running holdouts on more upper funnel brand forward video creative. Like I think you should actually— Start with those like directional tests inside of your hero channels. 'Cause how many, how many times have we talked about people expanding channels too early or like we don't think they're getting all the juice out of Meta that they can be getting? Like, I think you, you think you actually should start there to optimize Meta 'cause it's really hard to test all those things without a tool like House and then go into your like channel expansion, like YouTube, let's spend $10,000 a day versus no spend at all and like actually measure the the true incrementality of the channel as a whole. But if I could go back and start over, I would have waited like half a year or a year to do a lot of those channel holdouts. Because now that we've done all this marginal frontier testing to figure out like, oh, view content, super incremental for us. All these smaller things in Meta are super incremental. We have a way more optimized Meta account now sitting here today than we did a year and a half ago. But we didn't start there. And I think a lot of brands should figure out how to use house to maximize their hero channels and then go into like the YouTube holdouts or the CTV holdouts or the TikTok holdouts and kind of like do that channel-wide holdout. And then like, and then you're back into the, like once, all right, YouTube's incremental, now you're back into like the marginal frontier, like directional testing again. So I think that would, that's one thing I would change about how we did it at Hexclad is actually starting with the Like the, all right, 3 sales, where's the next dollar best spent? 'Cause I think those are some of the best wins and learnings we've gotten in the last year and a half. And like, I look at our account and we're spending millions on view content now. We never would've done that if we didn't have that like directional data telling us to do that.
What I could also hear is that you're almost best starting with a channel holdout of Meta, the same way that I just described we started at Ridge. Then you're doing all of these tests around tactics and figuring out view content works or video versus image. You guys have a great case study on the different incremental impact effective ad formats, um, et cetera, et cetera. And then you can kind of go back and say, okay, now with all these new learnings, we've got this new implementation, let's test Meta again. And you could measure that improvement. But even then, that to me now feels like a directional approach where like what I would care about is improving Meta. And at no point am I like, am I hitting my 2x incremental ROAS target? Because I think that, I think it's hard to do. But do you think you even need to start with the channel holdout?
Like if you're, if you're doing $30 million a year, And like Meta got you there. It's like, all right, we're 85% Meta, you know, 10% non-brand search and 5% branded search. Like, you know, Meta got you there. It's like, it's not, you know, do you even need to do that channel holdout? Can you go right into the, let's do like 30% more purchase conversion spend versus 30% more view content spend versus BAU spend? Like, can you just go right into that more directional testing? Do you think, Olivia?
I'll tell you why teams will often just come back to this is when you launch new channels, you need a reference point to compare against. Like if you're launch, let's say you're launching YouTube, you're launching some of these new channels. The next, the next question is, how does this compare to Meta? And so what we found is that like they end up actually coming back to the baseline Meta test because it is such a big it is such a big pocket of spend and they just wanna understand for these new channels, how does it compare relative to my investment in Meta? And then they understand how to reallocate. So, and then there's another benefit, which is that getting like a channel-level read can help tune your MMM. So we've seen a lot of like, I remember Cody did not start with Meta and then when he started working with us on MMM, we were like, hey, we're actually blind to like the biggest— channel in your business, and we just need to get a holdout on this because we need this for our MMM. So that's, those are 2 reasons why I find folks are, are coming back to the channel level read, um, is because they just like, they need a point of comparison for the other channels. But I'm, I'm with you guys. The optimization tests have been, and we saw this also with Jones Road, is like the, the, how do I improve Meta? Like the optimization tests, whether it's spend level, whether it's tactics, whether it's incremental attribution or view content or some of the mid-funnel stuff, like those are where the wins are happening. But it's sort of like Connor McDonald, you said this when we kicked off the conversation of your geo holdouts and your channel-level reads are a bit conservative. Like, I do wonder if, and this is, we see this all the time with bummer news, like A lot of teams come in kind of using Houzz as their report card of like, I need to go prove something to someone. And that is, I'm wondering if it's just a function of where you guys sit in your org, where you're, I mean, you guys are, you're not founder level, but you are very entrenched in these businesses. You have nothing to prove at this point. I think you've earned your stripes. Team where you have a, perhaps a CMO who's new, do you, do you like that? What you just described of like the absolute IROAS just feels conservative for all of these reasons. Mostly it's a 2-week test and we know there's probably compounding effects over time. Like, do you have any advice for, for, for marketers, like maybe in the position where they have someone to answer to and they're running these tests? and they don't look as rosy or as like, you know, optimistic as they want it to. And for that reason, like, would you recommend they just go straight to optimization and not even worry about the channel-level measurement? Just like any advice for teams on this.
Quick gut check for the operators listening. If you are spending on TV or CTV today, Can you actually say what it's driving incrementally? This is exactly why we work and have worked with Neon Pixel at HexClad for the last 3 years. We've grown with them a lot over time, and CTV has become one of our top growth channels. They help us treat premium living room TV like a real performance channel. It has smarter household targeting. It has suppression of people who already know us. It has a very robust analytics backend, so we feel really good about the measurement. And ultimately, and most importantly, it is a strategy that is built around incremental growth, not just claimed attribution. We look at incrementality as the true North Star on measuring channels, and they're really measurement agnostic. They don't force us into their own black box dashboard. They work inside the measurement systems we already trust and help us understand what TV is actually doing. If you want to check it out, go to neompixel.co and ask them to design a controlled CTV test on your numbers.
I think that no matter like where you start, whether you're trying to get that like the clear IROAs, we always say like raw IROAs, or you're opting just for the straight like directional sort of progressive approach. From a leadership perspective, it's best to just commit to this becoming an ongoing system of measurement. And even if you're just getting that that channel-level ROAS, which like you bring up a good point where it's like you want to run channel-level ROAS, like just to calibrate your MMM. It's like there's never— there are a million reasons why you would want to do that. At no point would I say that should be treated as a scorecard and that in and of itself someone should then decide I'm going to go spend less on Meta because this looks bad or my team is not good. So now I need to like go figure something else out. Like that's not really— I don't think it's, it's very rarely like the right way to interpret those results. So no matter like what your actual testing methodology is, the best thing to do is for the whole org to be aligned in like this just being a progressive form of measurement. And that is why if I were the CMO of Houzz, I'd be going on like podcasts and I'd be talking about this sort of thing so that like the industry can kind of come around on it because it comes up all the time. I think there's a lot, I'm sure you see it, I'm sure you've had this conversation a million times, Someone on board, someone tests house, someone gets one bad readout, and then they're like, what the hell am I supposed to do from here? And I think that comes down to like an education piece and a leadership piece of, again, just pro— uh, committing to that progressive approach.
I was doing some user research, um, on a totally different product that we're building. And one of my questions in that research was like, what are the biggest fires in your organization, in your role? And someone said, actually, the biggest fire is when we get a bad house result. And that just like broke my heart of like, oh gosh, like this is supposed to provide clarity and it's supposed to help. And when it, when, when we, when these teams experience bummer news, it's like a lot of my job is like coaching on how to think about it. And Connor Rowland, we were talking about this when we were together, like a lot of, a lot of teams with the right mindset understand the way in which a quote unquote bad test result is actually a great learning. And I want, I just, I want, it's, it's almost maybe a problem at the executive level. Like, I want these teams to have the trust and the buy-in to be able to celebrate the failures as much as, as the wins. 'Cause it's, it's really important and it's, it's, it, I, I think one thing you said, Connor, that, that's helpful is to think in terms of relatives versus absolutes is we've, we've come around on this is like the absolute incremental return is probably much less important than the relative performance of these channels, of these tactics over time and is long as we are moving dollars and reallocating from lower performers into higher performers, we are moving the business in the right direction. And you see that like Aaron Zaga from Newton, I just recorded a podcast with him. His bias to action was just like, it was so impressive how he didn't, he never like second-guessed a result. He just made a call and moved on and went to the next thing. And it just, no kind of like, you know, sacred cows in that like, oh, this channel, you know, there's no like kind of channel manager mindset where you have folks who have like specific kind of responsibilities. And I've seen the magic that happens when you do kind of just like look across the portfolio and you manage that way. And I, you know, I wanna make sure that teams don't get like stressed out about, the quote unquote bad result.
Totally. And the relative versus absolute is exactly what I was trying to get at when I say we've moved to almost an A/B testing framework where it's like, I just care about the results relative to one another. And then the other thing is I just want to be making slightly better decisions as often as possible. Like, that's the goal of the whole program. It's never like, it's never a scorecard. It's like, can we operate? Can we operationalize this? Can we get As many readouts as we can across the dimensions of the business that we really care about. And therefore, can we make slightly better decisions? And just understanding that like that progressive approach is going to push us in the right direction. And it's not perfect by any means. And it doesn't, if we're, if we're, you know, moving away from this idea of absolute, we're also never getting a clear readout, at least at Ridge on like, what is Meta doing every day? It's just a matter of like, do we feel that our media mix is slightly better than it would be otherwise. Um, and anyway, that's like, again, last year, I think we've really kind of turned the corner on that line of thinking.
Well, there really is no like bad result. I think we're, we're saying like, quote unquote, bad result. But as long as you, as long as you're allocating dollars better based on that result, like, is there really a bad result, quote unquote? Like, you know, the example, the most recent example we have related to what you just said, Olivia, is we were doing more non-purchase conversion testing on Meta. And we were doing the same 3-cell setup we did when we, when we tested against view content. And we saw that these, these 2 new objectives did not, they were not more incremental than our, than spending more up into purchase conversion campaign. Like you could argue that like surface level, that's a bad result. I would have loved to have unlocked, you know, traffic and reach and video views and all that and proven that putting more spend there is more incremental than more spend into purchase conversion campaigns. But at the end of the day, we still learned a ton and now we turn those off and we're putting more of our Meta ad dollars into a better spot because of it. So is it a bad result if you action it quickly? I think it's only a bad result if you like sit there and pontificate on it for a month and a half and you don't go and optimize your mix based on the data that you got. So it's like, what is a bad result? Well, the bad result is only a bad result if you're not going in. You know, it's like if you run a CRO test, but it takes you 4 months to implement the winning variant onto your website, like that's a bad outcome, even though you found a winning variant. It's the same thing with kind of sitting on your test results, whether or not the new thing you tried was quote unquote better or worse than the historical thing. I want to ask you a question because I'm curious how you educate customers on this. We talked about like doing the true channel holdout on Meta. And if you're a brand that's been spending on Meta for 2 years, 3 years, has gotten you to where you are, That's not actually a true holdout test, right? Because even the cells that you're withholding spend from, those demographics have been exposed to Meta ads for however long you haven't been running holdout tests. So technically, even though those cells are not being exposed during the duration of a test, they were exposed beforehand, right? And for a long time beforehand. So it's not like those cells, those demographics haven't seen your Meta ads. They just haven't seen your Meta ads while you're running this test. So Again, that just goes back to what we were saying earlier. Like even that is not a true net read of the channel, right? In reality, you'd have to run a holdout test for like a year or longer maybe to get a true net read. So do you guys educate your customers on that when they're running some of these early tests and it's like, oh wow, Meta only got us a 1.5x IRO? I was like, yeah, but the cells you weren't putting, you were not exposing spend to, like they were before this test ran. So how do you, how do y'all think about that? And like, Just making sure your customers know and like are aware of that kind of phenomena.
Yeah, that's a great question. I've heard you guys talk about this before. Uh, so you have, let's say, you know, you're spending nationally, you've been spending on Meta for a decade, and then we are running a holdout test and you're saying that holdout group that we're, where we're turning off ads has been getting ads for 10 years. So like they're not actually truly held out.
Most brands treat customer support as a cost, but with AI, every support conversation is now a chance to increase retention, recover revenue, and grow your LTV. Richpanel doesn't just use AI to answer tickets. They allow operators to run revenue experiments inside your support. You can test different win-back offers on refund requests, subscription save offers during cancellations, and product recommendations that turn chats into sales. Plus, they guarantee you're going to save money. 50% of your support volume automated in 30 days or your money back. That's a Pretty good offer if you ask me. So go to richpanel.com/demo. They'll build an AI support team live for your business and show you how it can drive incremental revenue for you.
The reason that's okay is because that dynamic is happening in both the holdout and the treatment where like those people who have been exposed to ads for a decade, they exist in both the control group and the treatment group. So what it's doing is like it's inflating the baseline Of both groups. And so then what you're measuring is the incremental effect of this new treatment, the treatment group getting ads for a short term, like for these, you know, 3 weeks or, or whatnot. So the long-term effects of the users who have been exposed to ads for a very long time, that is present, that is happening, but that's happening in both the holdout and the treatment. So it would just kind of like lift up the baseline of both groups. But what you're not answering, I mean, again, what you're not answering in that question is like, what are the long-term effects of ads? And this is why I've been talking a lot about like, if the hypothesis is that these ad effects compound over time, then that is a reason to run a longer test. And so that's come up with like, you know, the more upper funnel channels, like a CTV. Again, it's just kind of aligning the test design to the hypothesis. Like if you believe that there's something happening around compounding effects over time, then you need a longer test to measure that. So that's, that's how I would think about it.
And that's been our approach at HexClad with a lot of the testing we've done on like YouTube and CTV. Like we are running, I mean, we're exposing these cells to holdouts for like 2.5, 3 months sometimes with an observation window that can last 3 to 4 weeks. And It's kind of painful to let those tests run that long at times. Like, I wish we, at times I'm jealous of the brands that have, you know, like a $60 first order AOV and their, their like sales cycle is a lot shorter than ours. But at the same time, it's given us so much confidence to scale up YouTube and CTV and some of these other channels that I just don't think there is. I think there's a lot of ways to kind of suss out incrementality without a tool like Houzz using you know, like GA4 or a server-side analytics tool or Northbeam. I think there are also a lot of tools, a lot of channels that you just literally cannot do it without a tool like House. I think CTV is a great example. And now we are, it's, CTV is one of our fastest growing channels in the last 3 years. No way we would've been able to have that level of confidence without like a, like a 3-month holdout test with like an observation window that ran through a sale. Like we just have to do it that way. But not, that's not everything. Like some things we don't have to do it that long, but like CTV and YouTube channel holdout tests, like the classic example of, um, you gotta run those things for a long time and just be patient. Um, I think you're gonna get a lot better, more actionable readouts if you can be.
Connor McDonald, you, 'cause you mentioned you guys run 2-week tests. How do you think about the reality that different channels have different payback windows? So for like a YouTube, for example, are you comparing your 2-week YouTube readout to a a 2-week Meta readout? 'Cause this really trips up customers of like, yes, they're on an even playing field in terms of length of test, but it might be the reality that just certain channels have different dynamics in terms of payback window.
We run, we mostly run 2 and 3-week tests. We are just about to launch a test today that will be 6 weeks. So, you know, we're getting there. We're coming around to a, uh, month-plus long test. We are looking at those. We are looking, this is a, this is an example just of like how we ran this test. We ran Meta and YouTube. We measured both of these. We looked at them directional to one another. We were not thinking about like, well, what does this mean over the next 6 months or something like that? Like, what is like the long tail of YouTube value? That might be because we were optimizing for relatively short-term outcomes where we said, we really want to figure out how do we best deploy the next dollar for the week of BFCM. And then it's these longer tests, like the 6-week test. We go live with our sweepstakes today, which is super exciting. We have Tony Hawk that's a part of it, which we're super stoked on. And So we're going to do a full 6-week test on some of the upper funnel channels around the branded Tony Hawk content. And the point being like, what I would like at the end of this is a data point that we can point to to say like, is this type of partnership, is this type of content, is this media buying strategy, like, does this have a larger impact over a longer period of time? So it really just depends for us on how we want to be looking at it. We have examples of both. We do err on the side. I talk about it all the time. We do err on the side of shorter tests. more frenetic pace in terms of what we're testing. Um, and then we knowingly are making some compromises when we're not considering the total long-term impact of some of these different channels. Um, but there are some exceptions to that as well.
What I appreciate about your approach sometimes is like, you don't overthink it. Progress over perfection. Like you could find places to poke holes in all of these approaches. So, um, you know, sometimes it's just, progress. Can we talk real quick about, like, one of the big challenges is on operationalizing, like, a system around incrementality? We've been thinking a lot about this at Houzz. We've been adapting our product roadmap accordingly of, like, geo test, moment in time snapshot, some of the challenges that you voiced. How do we better connect these tests in the measurement back into kind of like buying decisions that you need to make. So, I've been thinking about this as the macro and the micro decisions that need to be made. So, at the macro level, this is like channel budget allocation, like where am I going to allocate budget at the channel level? Maybe this is weekly or monthly. A lot of what we've been doing is like building this into the MMM where You can do this. It's like kind of clunky to do that only with experiments where it's like, I have this Meta read and I have this other read. How much should I move from one channel to the other? And so the MMM is solved for that, where there's like a nice slick UI scenario planner where you can like ask what-if questions. What if I reallocate $1 million from Meta into YouTube? The MMM also handles non-geo-segmentable channels, which is tricky, like affiliates and podcasts and whatnot. And then you get these like really nice clean return curves of like exactly how much budget should I move? Whereas that was just kind of fast and loose before. And then on the micro level, this is new. The next question from there is, okay, then what campaigns do I put that, do I add that budget to? Like within YouTube, like where do I add that budget? And so this is why we have built a pixel. It is now live. You can add the house pixel. And what we're doing is we're, we're the value. So much of the value of attribution is like the real-time stream of data. Like, we need to know what's going on very quickly, you know, at a granular level, even more than daily, hourly. And so, by having that, that real-time stream of event data, we are, we've, this is, this has been like 2 years in the making. We've built a machine learning model where we're trying to like de-bias the attribution data with our experiment database. So, they have like, you know, thousands of experiments. We're debiasing that attribution data based on the cross-customer database and then your own experiments where you've run them. And that is our way of saying like, okay, so what campaign do I put this into? We just historically, like, you kind of use a different system to figure out how to do that. But a lot of our customers have said like, well, I wanna make that decision also like rooted in incrementality. And so, that is like together, I think the combination of the more macro and the micro is what's helping these teams kind of operationalize. My next question for you is like, should a human be doing that? Like, so like that, just think about the, you got experiments, you got MMM, you got attribution. We are attempting to solve the triangulation problem of saying like, here, this is all kind of calibrated in causal truth by your experiment data. But like, do you wanna log into different views and try to like, kind of, you know, uh, do you want to do that as a human of like triangulation, or do you think a system should be doing it for you? Enter agentic media decisioning. Thoughts?
We look at House as like this outer ring of measurement. And then like after you, you get the, the readout on this outer ring of measurement, You're trying to create a relationship between that most outer ring and a more inner ring that you're using on a more day-to-day basis. So like the easiest example that we talk a lot about on the podcast is like creating these multipliers between different measurement channels. So it's like, hey, we know that for the duration of this test that we ran on view content, we liked the IRO as readout because it was more efficient than the same amount of spend in a purchase conversion campaign. Great. And then we're saying, okay, like now how do we take that data point and, and like to your point, Olivia, operationalize it in our daily, weekly, monthly decision-making? Well, let's create a multiplier. We know what the IROAS was during this moment in time. We can also go into whatever MTA you use and say, well, what was the ROAS or CAC or whatever, like in-platform MTA metric you care about? And now we create a relationship where we can say, okay, if, if we're at this in Northbeam, we know that that maps out, at least in this moment in time, to this IROAS we were comfortable with. And now that becomes the number that we're moving budget up and down on. And then maybe every once in a while we'll rerun a holdout test to kind of recalibrate what that like ROAS or CAC number should be in Northbeam. And you could do it with, I mean, I think there's a, that's just one example. I think you create a multiplier with a lot of different data sources. Now, should a person be doing that or should a robot be doing that? I think like, yeah, over I don't think we're there yet. I think it's very much the same as trying to take like an American UGC ad and say, hey, make this Italian and like get it in the ad account. Like it's, um, I think we will get there. I just think there's a ton of, there's so much context, right? That you would just, I think the role will turn into, great, how do we take this context and like give it to this agentic media buying? Like how do we make them know that a 0.2 one-day click ROAS is actually winning an incremental according to our holdout test? And then how can we like, set up rules to make that machine move budgets up or down based on that. But then times what, hundreds of tests and like hundreds of like nuanced pieces of context. Like, I think that's where the complication comes in. But I think we will get there. Yeah. And I think ultimately it will be better than a human buying that media and like doing that triangulation themselves. But that's how we're doing it internally. I think we're fairly far away from being like, yeah, like let's just let like a, an agent buy our media based on all these rules. But I do think that's where we're headed. And I don't know when we'll get there, whether it's in a year or 5 or 10, but I do think we will get there.
Yeah. The, the 2 big qualms that I've heard when we, when we talk to brands about this is it needs to understand all of my business context. Like, cool house, you have a lot of our data, but like, hey, we're looking at post-purchase data. We're looking at our promo calendar. We're looking at our product launch schedule. And my response to that is like, cool, like we can take that. Like that's, that's part of our MMM already. Like we're already ingesting a lot of those as covariates. So like that feels solvable in terms of like, how do you get the system enough business context such that it has everything it needs? And then the other big qualm and objection I hear is, Like, it needs to be explainable. Like, you need to be able, like, it needs, there's almost like a sense of safety of like, if I'm gonna hand over the keys to the system, I need to understand why it is making these decisions. But that's also like, that is sort of the transparency around the experiments is what's helping us. Like, it's sort of what's helped us with the MMM is like, you can see a beaker and you can see exactly where an experiment is informing the call. So, like, those feel— Solvable short-term. Um, Connor McDonald, like what, I guess I, I just, I'm gonna, I'm gonna keep pushing on like the, the why, why is this, why, why can't this be done, uh, by a system?
People are the most important part of your business and finding real A-players is hard and incredibly time-consuming. That's why I wanna tell you about Propel. Propel is honestly the best recruitment agency that I have personally worked with. They are marketing and operations specialists. They source talent from US, LATAM, and beyond for top e-com brands like Jones Road, like many others that you know and love. At Jones Road, we've hired over 12 people through them. We've hired video editors, graphic designers. We have multiple retention managers, creative strategists from them. What I love is not only are they easy to work with, they're quick, but they work on performance. So you only pay a one-off fee only once you hire somebody. You don't have to pay, you know, any retainer, no ongoing fees. Plus every hire comes with a 3-month guarantee. So I highly recommend them. They've worked really great for us. We have a really great team that they've helped us source. Head to the link in the show notes and submit the form to get 10% off of your first hire.
I really like the phrase debiasing. I think that's really good. And that's like, that's a great way to try debiasing the MTA data is how I interpreted that. We're like with the pixel, we're seeing a certain result like ROAS. And we're debiasing it based on all this additional information we have via incrementality tests. Uh, that is theoretically what my team is trying to do all the time, where we say, okay, we're getting this result from this meta campaign and it's a view content and the ROAS is really low, but based on all these previous tests, we're going to debias and say we're comfortable spending on this campaign. But would you say that's the same? Uh, Connor brought up, um, developing incrementality factors. It's really just like, is debiasing like more granular incrementality factors?
And it's automated.
Automated.
Automated.
Right, right.
It's the next, so the, one of the learnings we had from incrementality factors is they work well, but they're very blunt instruments. So, just like a lot, very blunt. Um, so, the model, uh, takes into account the, the model that's doing the debiasing, it's automatically applying these incrementality factors, but there's a lot more that goes into the factor than just your single experiment, the most recent one. It's all of your experiments that you've run over time. you know, recent ones are weighted more heavily, but also moments of seasonality that are similar to today would also get weighted. And then cross-customer insights, it's also your investment levels. So, there's just like a lot more that the model is tuning on than one single, like, static experiment result. But yeah, you nailed it. That's exactly what I meant, I meant when I said the attribution product is like debiasing, uh, is we are doing kind of like a souped up version of incrementality factors in an automated, automated way.
And this makes total sense to me. And like I said, it's what my team is trying to do all the time. And you can imagine it, you can imagine how far we are from being perfect in that, like, like to a crazy degree. I mean, I just, I've mentioned earlier, we run all sorts of tests. Like we see, um, significantly different incrementality factors between men and women within our wedding bands.
line.
We see very different incrementality factors between view content and purchase optimized within Meta. We see a difference between video and image. So like all of a sudden, if we are running, you know, video view content ads to women on wedding bands, like we're supposed to be like considering all this context when we decide, are we spending more dollars on this campaign or not? We are not even close to being perfect with that. Like, and I know that for a fact, like the only thing that I'm somewhat comforted by is I know that we are at the very least working with more data than the average brand. And I think we are considering some of it when making these decisions, and therefore we are making slightly better decisions. But if you told me and said, hey, we are gonna consider all these tests that you've run, all these different dimensions and, and attributes of the ads and some of the other things that you mentioned in terms of debiasing, there isn't a doubt in my mind that a system could do that better than a person. That's like my general answer here. What I will say, just to hit a couple of the other ones that we, that we, uh, hit on, you know, this idea of like Geolift calibrating MMM and then doing more granular optimization with MTA, like makes total sense to me in like a scientific way, an economics way where it's like, yeah, like if you do the model, like that all makes sense. And then it does, it feels like historically there's always been like more nuances in terms of context and things like that. So I guess that's where I would start from an economics perspective. Economics treats itself like a science. I got an economics degree and it's like, yeah, you don't have rational actors working in it. Like the models never play out the way that you anticipate. in the way that like a physics problem might. So I've always felt like we've had much, we've been working in a scientific way, but with much sort of blurrier lines. If we can begin to, you know, account for some of the additional context and the debiasing, I think that goes a really long way. And then, yeah, the other thing that I would say my team is doing often is looking at things like inventory. Like our promo calendar. Um, like, you know, there's this, there's something, and this is maybe the one point, actually, I'll say that quickly here, the promo calendar, um, the business objectives, we can instill that. And I say that with like Meta, we talked about this at the Meta Performance Marketing Summit. We'll always set up our ad accounts and our campaigns in a way that aligns with the way that we want to sell our inventory. Like that's what we're doing as a business. And I imagine House and anybody trying to approach agentic media buying will have to accommodate some level of like guide that marketers have to put in place because we can't just operate this like Meta treats it this way, this like economic experiment where it's like, hey, just give us your ads, give us dollars and we're going to spit them out on the other end. Like that, that never works. So it's like just a matter of sort of containing some more of these constraints. I have no issue with that. That seems like it's, it's pretty clearly going to be adopted from like a, technology perspective. And then the only one that like maybe I have some sticking point over is like long-term business objectives where we might have some sort of very specific, um, goal for a very certain type of product to the degree that like we're actually going to make a decision around it that is, um, in conflict with the way that we otherwise would. But like at this point, like I'm really getting down to like a very fine detail. So my answer to your question is like, It feels like we're not that far away from a really large percentage of what we consider marketing or media buying or media allocation being done far better by a machine.
And you, you nailed it with like what your team is trying to do. It's, it's about the signal, right? It's, I think it's why you might not trust like a Meta agent to do budget allocation is because you're like, well, it's, it's using its data. It's using like platform data and I don't care about platform data. So this is, I think like if you trust this signal that's powering, like the foundation that's powering it, then this becomes conceivable. And if you think about like financial portfolio management, like this is already happening. Like why doesn't this exist in marketing? And I think you said like, you know, sometimes we're not all rational actors. Like there are these things that are gonna happen in terms of business nuance. You know, you've got the, the CEO who wants to see billboards on the 405, whatever. And this is what we've done is we've worked that into the system of like, what are the exceptions that we need to work into this system that are specific to your business and that nuance? But we're spitting out and we're not claiming to automate media buying. There's so many components of this. The part we're starting with is budget decisioning, and we're spitting out these recommendations, and we have brands now like approving, rejecting, and then there's the next step of like autopilot, like, do you want us to go implement this for you based on APIs into the ad platforms? And we're not there yet. It's still like more of a conversation before the implementation happens automatically. But we're starting to see like, like probably a handful of brands accepting these recommendations, but it took a while to get there in terms of trust and just like, does this have enough context to make the right, to make the right call? But it's pretty cool that it's, it's happening. I think it's moving in that direction and I just don't hear a lot of good reasons why this can't be automated.
Does the model still work if you have like a handful of channels that you've run a bunch of holdout tests on, but then you still have some over here that you haven't? Like, let's say you've ran a bunch of meta tests, a bunch of like intra-channel meta optimization tests, like static versus images, view content versus all these things. And then like you've done some testing on other channels, but let's say maybe you've never ran a holdout on like Pinterest, TikTok, YouTube, but you have on like CTV. Like, does the agentic recommendation still work if you've only ran holdouts on some of the channels, but not all of them? And is that where like— the business context and all the other data sources comes in. It's like, all right, we don't, we haven't ran a holdout test on CTV, but we do have your survey data. We do have the in-platform reported data. We do have like UTM data from the short link you're overlaying on the creative. Like how does that work in a tool like the one you're describing, Olivia?
We, you know, we, to your point, like we have the pixel data, we have the, we have a lot of this data and then we have the cross-customer experiment database as well. that's informing the initial kind of like, we call it like cold start estimate for a channel that you haven't tested. And then, this is what I, this is my favorite feature of both our MMM and what's going to be built into Architect. This is the name of our agentic decisioning system, but we show you where we have an experiment to inform the recommendation, and we also show you where we don't, and then there's like a confidence interval associated. And so, There's this nice flywheel where it's like, if you don't trust that recommendation, like, we get it, maybe you should run a test first, you know, because we're like, here's what's informing this recommendation, and if you don't feel like there's like robust enough evidence, then that's a really nice flywheel back into your testing roadmap of like, okay, then we're going to go test this next. But the incrementality index, that like collection of experiments across customers, is also helping to inform some of this as well.
Oh, okay. So your aggregate data, all the, all the other tests you ran on CTV for all the other brands can also help the brand that's never ran that CTV holdout test.
Exactly.
Very cool.
Before Q4 hits, the smartest e-commerce operators aren't just asking, how do we spend more? They're asking, how do we make every visitor more profitable? That is where AfterSell comes in. AfterSell helps e-commerce brands increase revenue per visitor, contribution margin, and incremental profit by optimizing the highest intent moments. cart, checkout, post-purchase, and even the thank you page. Brands like HexClad, Ridge, and Pela are increasing their AOV across every cohort using these zero-risk upsells. We've tested and they have no negative impact on conversion. It's just more free money in your pocket. And with Rocked Thanks, you can offset your AfterSell subscription. No redesigns, no risky funnels, just better economics. Wanna see how much money you can make with AfterSell? Go to aftersell.com/rocketrevenueforecast, put in your numbers, and see how much more profit your funnel will make before BFCM starts. We started our, like, I don't know, it was probably 4 or 5 years ago at this point where Ridge wanted to approach like better forms of measurement because we were running a really low MER and we were getting to the point where it's like, if we can just find the wasted spend here, it's going to be like extremely additive to the business. And we started with MMM, which is what you're describing, which without theoretically without any geo lift tests, the MMM will try to figure out the, the, the ROAS of each channel. But like what our experience was, was that we just had these really large confidence intervals and you could run these weird experiments where you scale up and you scale back down really quickly and it's going to try to tease out like what was the effect on revenue and therefore what's the effective incremental ROAS of this channel. We just found we had these really big confidence intervals. So what you're describing, Olivia, is one, the, um, the cross-customer data that you mentioned better informs what CTV might be doing. And then also running Geolift helps. Geolift tests can then calibrate that model and directly inform it as to like, what is this doing for your brand? And yeah, I think that's an exciting place to be. I always joke, um, we don't use an MMM currently. We use my mental model, which is just like, we're, we're taking in, I'm taking in the data and we're making some gut decisions over here. Um, and that's really just because we've had a, we've, I think we have an odd business with these 4 different product lines and and channels within each of them. So it's always, we haven't had a great tool for like the bespoke nature of our business. And I think we're also getting to a place where MMMs 5 years ago, I think we're like kind of one-dimensional. It's like, hey, we're going to take in revenue. We're going to take in channels. We're going to model them out that way. And I was like, our business just has a dollar spent on Meta can be spent in like 8 different ways. I'm like, I just don't think we're going to capture the nuance that we need to. And it feels like, um, the tools today are just becoming more bespoke, more customizable. We can set it up in a way that better aligns with like the dimensions that we care about within our business.
I understand why MMMs get a bad rap. Like I, one of my big learnings over 4 years of Houzz is like geo experiments are the best we have. There is a counterfactual, there is a holdout, and there's still noise. Like marketing data is so noisy. It's so messy. Like, you know, the amount of conversions relative to the amount you're spending, it's just It's like very hard to find signal. And so why would you expect an MMM to be able to do that? Like, so that's what just, I'm like, I don't need, like, there's, based on what I've seen on confidence intervals around an experiment, I don't understand how you could not have just gigantic confidence intervals around a model that is not informed by an experiment. Because there's just so many, this is what our scientists will say is multicollinearity. There's just like so many confounding variables at any given moment. Like you are dialing up all of your channels at the same time you are starting a sale. Like how could a model actually disentangle which of those things is causing, uh, the sales? So yeah, MMMs are, it's, it's tough, but the, that, you know, of course the best we can do right now is we can debias with, uh, with experiments.
I like the debiasing term a lot. I hadn't heard that and I'm like, that's a really, that's a really good, example of what everybody's trying to do all the time. And like, we're all trying to connect these dots and just doing that in a more automated way makes perfect sense to me.
You know the difference between hitting your numbers and missing them? Clear signal on what's actually driving growth. It can get really, really noisy. There's so much noise. You got platform data, you got blended data, MMM, all the acronyms, MTA, experiments. All of it, all pointing in different directions. The more you're spending, the faster you move, the more bad signal can cost you. That's why we use Houzz and we've been using it for years. That's why the other marketing operators do as well. They're the best tool on planet Earth for measuring what we call incrementality, which we talk about a lot on the podcast. What is the true impact of your advertising dollars on your business? We have causal MMM for channel-level budget calls, causal attribution down to the ad level, and Architect, their AI agent, tells you exactly where your next dollar should go. And the results speak for themselves. StockX saw a 41% lift in IROS using House, and you're not stuck with a help desk. You get an embedded measurement strategist who actually helps your team make better decisions. Their whole team is great. We've worked with a lot of them. They are world-class there. Go to house.io/operators, H-A-U-S dot I-O slash operators, and start backing your budget calls with real causal data.
I've got some over-under-rated questions for you.
Yeah. I've heard you do this twice now and you need to make it a regular segment. It's my favorite. It's my favorite part of the episode.
Yeah. I'm glad you like it. Yeah. Connor and I tried it a couple weeks ago and it's like, oh yeah, this feels, this feels good. It's a great way to get through some stuff. We'll, we'll, uh, all right. This came up earlier. The word folks.
I, yes. When you were talking about waving at the end of Zoom calls, I was thinking about how some of my, like, I joke, like my real life friends, who are not in our industry. They're not like tech business people. They make fun of me for that. They also make fun of me for using the word folks. I'm like, but I don't know what to say instead when we're like, do you say people? Like, we're seeing that folks are implementing this. Like, I think people sound sort of rude, so I don't really have a good alternative, but they're like, you've changed.
Folks does not strike me as corporate at all. Like, I think, like I said, I think it's, I think it strikes me as like sort of Midwest and charming. I'm like, yeah, that's cool.
Folks are doing it.
I'll go underrated on folks.
Like, that's all folks. Sounds kind of, it sounds kind of cheesy. It's pretty cheesy.
When you use it that, that's an extremely underrated form factor of folks that you just used, Olivia.
Connor Rolland wouldn't use the word folks. You can just tell.
I don't use the word folks. I, I, I'm more of a guys. Hey guys. See you guys. What's up guys?
Gendered terms. Connor's just sending it.
Yeah.
Slack bot.
The Slack bot is gonna. Slap your hand.
I was actually asking a friend this the other day. I'm like, is guys— has guys gotten to the point where it's not even gendered anymore? I was asking some girlfriends of mine and they're like, no, I think guys is like a fair way to address a group of people in a non-gendered way. It's like this— it's like a separate meeting of guys. Do you agree or disagree?
I think we're reaching that point pretty quickly. Not a lot of good alternatives. We have folks and we have y'all.
There used to be a Slack bot, like a corporate Slack bot that would alert you, like when you, it would say like, this is a, you know, it would, it would kind of like hand slap you when you use the word guys. So I've been, I've been trained not to use it.
Yeah, I was wondering that if I should not, if I should not use it. I got the, I got the go-ahead from, from some girlfriends of mine, but may— now I'm getting, now I'm getting a signal pushing me in the another direction. You're making me question my decisions. Maybe is guys, guys over underrated? I think maybe it, maybe it might be overrated. I don't know.
Maybe it is. Maybe y'all, y'all, do you say y'all?
Uh, I try not to, but sometimes it slips out.
You also have dude, and that's, I think it's not fair to women. Like your version of um and like is dude. Like I, I hear Sean, Sean in, in the pod is always saying like, dude, dude, dude. And it's like, it just comes off as way cooler than an um or a like, but I can't say dude.
So why not? You can say dude.
No.
I feel like dude's kind of like guy. It's like, it's like, it's not even gendered anymore. Like you can call anyone dude.
Just decoupled from, from the original meaning of the word completely.
Right, exactly. Like the same way guys kind of is decoupled. It's like you're addressing a group of people, guys, or like you can address anyone as dude. I think, I think you should start using dude, Olivia.
Uh, all right. I got, I got 2 more. One's a, one's a real one. And I put this on the list actually, 'cause I didn't realize you guys were doing it. Taylor Holiday talks a lot about using the collection of Geolift studies to like pre-inform or like, quote unquote, like assume what the incrementality factor might be for a new brand. I wanna say over-underrated on that, 'cause I wasn't sure what your perspective was gonna be, but it sounds like you guys are on board with that.
I think it's extremely helpful for the cold start problem where you don't have an experiment of your own. Like what we've seen is these brands really are different and the same channel, the same tactic might, You're seeing this with incremental attribution right now, where some brands are like, this is absolutely crushing, and others are like, this tanked my business. So, we do see like that in the hierarchy, the brand-specific results are extremely important, but in the absence of that, for cold start, it's been extremely useful. So, this has been a huge unlock for us on our, just across our entire suite of products. So, uh, underrated.
Underrated.
Yeah, I like that. Can I ask a clarifying question on that, Olivia? Are you doing, are you controlling for like AOV bands to make sure that like a meta holdout for a $175 first order AOV is not being applied to like a $30 food product or whatever?
Yeah, I mean, this is, it's why it took so long to build. It's why we have our best people working on it. This thing is so We've been working on it for 2 years. So that's, um, I think it's, it's just, it's been very, it's been very complicated to make sure. What we do is we, we take, it's called the incrementality index. So we take like the prediction for any given channel, and then when a customer runs a test, we look at how the test result compared to our prediction, and we call that a win rate. And to get the win rate up to where it needs to be took a lot of tuning based on all of those factors.
So it's almost like your version of like an MMM's out-of-sample kind of, right? You're basically saying like, did the actual holdout we ran get close to what we predicted it would be based on all of our aggregate data? The same way an MMM's like, hey, we're going to stop pumping in the data from June 1st, and we're going to see based on how much you actually spent, if our model predicted what we know to be your actual revenue. It's kind of a similar philosophy. Yeah.
Yep, exactly. And then you just, you have to be really careful with like test labeling. Like a lot of customers run spend-up tests and you have to treat those very different. Like when you're just testing the marginal efficiency of spend, like that needs to be labeled correctly when you set up an experiment. So it's like a lot of cleaning around like making sure that doesn't contaminate our channel-level reads for—
What did you say about naming conventions being really, really, really important?
I wrote that down during Meta Performance Summit. Is, uh, Connor, you had an awesome quote on stage about the importance of naming conventions, and I totally agree.
They're the heart and soul of any brand.
Heart and soul. Uh, all right.
I got the last one. We'll close out on this. Over, underrated on the measurement strategy team at Houzz, and I'll be the first to say extremely underrated. I wanna give a shout out to Noah, who was on a call with my team yesterday for like an hour and a half. It felt like going in depth on what kind of tests we wanted to run for our upcoming sweepstakes. So, um, I think we're probably unanimous, underrated on, on the measurement strategy team.
I've worked with a lot of good teams in my career. Like I've, we, I used to look back on the Netflix growth team as like, it just does not get better than this. And this is one of the most impressive teams and they make each other better. There's like this spirit, you know, where when you, you like don't wanna let each other down. Uh, and so they, I mean, when we hired Nick, I thought we like, you know, we have Nick and Alyssa were like our founding team members and I, I just thought they were amazing. And then like, They hired people even better than them. It's just been like so, so impressive. Noah, we just promoted Noah. He's crushing. He'll, he'll still be around, but I, they, yeah, this, this obviously this team is like very near and dear to my heart and yeah, very, very underrated.
I, I was gonna say properly rated, 'cause I feel like everyone looks at the House team as like, they're like an A1 CS team.
I was joking. I have never cried in a work setting. I can't imagine crying in a work setting except when Nick Doran told me he was leaving to, uh, to start his brand. I like, I was like, I gotta get off this Zoom call. The tears just—
That's why, that's why despite being like, it's an open secret that it's like just an extremely talented team, it's still underrated. Olivia's crying when people leave. I mean, it's like you can't, that's not properly rated.
Yeah, no, I just wanted, I'll end by just saying we talk a lot about this internally, and I don't know if I've ever properly expressed to you, like, thank you for how much you've done for this community. Like, I feel like when I talk about how that first chapter is, is making sure, you know, people know that incrementality matters, like so much of this is, uh, what you do on the pod, you know, we, every week. So, thanks for, for everything that you've done and, uh, and for, for pushing the industry forward.
Oh, appreciate that. It is a symbiotic relationship though, because without incrementality, we'd have had like 20 episodes. So we're about 100 and something more than that because of you guys. So, uh, yeah, no, goes both ways.