The reality of automated valuation models and AI trust

Allie Barefoot: I'm Allie Barefoot with Cotality and this is Data in Context. Today we're diving into AI and home valuations. The value of a home is now decided by AI, but the biggest challenge isn't the technology—it's trust. Cotality's survey shows only 22% of Gen Z home buyers trust their mortgage broker. If they don't trust the person, how can they trust the opaque AI setting the price? This transparency gap is exactly why federal agencies issued the new AVM Final Rule, mandating that lending models be unbiased and auditable. But how does this advanced AI actually show its work to ensure fair valuations? Well, to break down that data, we're going to be talking with Saima Jewett, Principal Product Manager of Property Intelligence at Cotality, to explore what it takes to build trust in home valuations. So, let's go on ahead and jump into today's questions with Saima. Saima, thank you so so much for joining us here on Data in Context.

Saima Jewett: Thank you, Allie, for having me.

Allie Barefoot: Yeah, of course. This will be one of many conversations that we're going to be talking about AI here on the Data in Context channel. So I kind of want to just set the scene here for a lot of our listeners because in October, federal agencies issued the Automated Valuation Models—typically known as AVM in your world—and they announced their final rule. Can you explain a little bit about the objectives and policy of that final rule and how this kind of plays into the use of AI?

Saima Jewett: Yeah, that's actually a great question. I do want to start off by saying that we continue to follow the 2010 Interagency Guidelines. And since the new ruling which came out in October 2024, which is not much different from the 2020 one that I just spoke about, it emphasizes on points, of course, which, you know, are already that we are following internally here, which is—there are five of them which I'm going to go over. It's ensuring a high level of confidence in the estimates produced, protect against dynamic manipulation of data, seek the avoidance of conflict of interest, require random sampling of testing and reviews, along with comply with applicable non-discriminatory bias and the laws. As we continue to follow all these rules in helping our clients and providing them with model methodology documents and performance documents, we continue to state and we continue to say that we test our model for bias. We provide accurate and reliable valuation with robust testing and maintain those standards with ongoing monitoring on our AVM products. Now, in addition to that, our extensive property data and our advanced technology of AI, which you are referring to, is one of the sub-components of the product Total Home Value. But it does not change how we run the model. It's essentially a mathematical, rule-based model that is monitored on a rigorous internal basis oversight internally. As we continue to assist the clients by advising them to send in their own files, they have the capability to review and test, you know, this data internally for audit purposes, and we also continue to participate in our third-party independent testing and provide our clients with the internal blind testing results that show our quarterly performance of our model and how it's going, you know, on a continuous basis. So yeah, the gist of it all is that's what we are, you know, doing at the current moment.

Allie Barefoot: Well, I can't say you're busy at all, my goodness!

Saima Jewett: Just a slight.

Allie Barefoot: Right, just a little bit. [laughs] No, it definitely sounds like you have your hands full, but it does sound like you and your team are definitely keeping up with the models that are continuously coming out to help that prospective home buyer or current homeowner. And we did a survey here at Cotality that shows that only 22% of Gen Z home buyers, they trust their mortgage broker. So building trust is more important now than ever, especially with a lot of technology coming into play, which is why confidence in technology's accuracy is so crucial. Do you think that this new ruling is a step in increasing transparency and trust?

Saima Jewett: Well, let's talk about Gen Z here. I will say that that 22% stat from Cotality is incredibly telling, right? But I'm honestly not surprised. We must remember that Gen Zs are digital natives. I've got a Gen Z in my house—two of them. They know better than anyone else that that algorithm that we have is as good as the data that you feed it, right? If a computer spits out a home value that creates a mortgage payment that they can't afford, they want to understand the methodology on how this home value has arrived for them. This is exactly why this AVM, which is a rule-based model, can step forward in that trust process. Now, the rule also mandates that we do quality control and random sampling. What this does is this forces lenders to essentially work with their data providers, such as Cotality, where they submit a file to us on the random sampling which I discussed and, you know, mentioned in our previous question. It addresses and makes sure that the information that they're receiving on the property is adequate and they meet their audit and valuation threshold guidelines, because that'simportant. They want to know, they want to understand, and they want to make informed decisions versus rash decisions.

Allie Barefoot: And you said that they want to be informed, right? Nobody wants to be left in the dark. And there have always been questions around the bias of AI, typically just what is AI, but largely this has a lot to do with the lack of transparency. So how do models like Cotality'sTotal Home Value manage to work its accuracy and prove its accuracy to meet those new AVM model rule requirements?

Saima Jewett: Yeah, that's actually a really good question as well. I know there's a lot of—there's a lot of nuances, and the AI word has been thrown in quite a bit. I think there's a big fear with AI that it's a big black box where the data goes in and a number kind of pops out. As this new rule continues to develop, you must think a little bit differently. This is why we built the solution Total Home Value to the power of X—a little bit differently than all our previous legacy models. We don't just want the model to give us an answer, right? We want to see and understand how it performs in real-life scenarios. We do this in a very specific way, which goes above and beyond the standard requirements. First, we use the blind testing methodology. We continuously feed the model with the new sales data which the model has never seen before to ask a value for these homes. This proves that the AI just isn't overfitting data to look good; it's understandably saying, "Hey, this is the market value." Secondly, we publish a confidence score for every single valuation that's on the property. So it's crucial to have that transparency, that our model just doesn't say that the house is worth half a million dollars due to the accuracy and the validity of the data—it vectors if the model is 95% accurate or 70% accurate, and it has the confidence based on the comp selection within the area itself, along with, you know, where it resides. Where is that property? Is it in the rural area? Is it in the most populated area? So it has that intelligence to kind of detect all those parameters. Another point to also note as well is that we continue to assist our clients to advise them of that continuous testing and model methodology. We continue to participate in our third-party testing, as I've mentioned before, that clients can engage for services. We also provide these internal blind testing results on a quarterly basis to ensure that those accuracies are met when the new AVM guidelines continue to evolve and we get those. We also provide model attestation. I know there's a lot of concern about that bias that we talked about. This attestation pretty much states that our models do not contain any kind of bias. So yeah.

Allie Barefoot: That definitely allows that transparency to come through and show a little bit more trust, for sure, when there is no bias involved. And I want to talk a little bit more about a special feature you talked about, you know, evaluating homes and what goes into that, but smart features. They do add a lot of value to homes, but they are changing rapidly—new upgrades, new updates. But they're making today's technology quickly become obsolete. If smart features can rapidly lose value, and AVMs can't always track this data from public records, how can a giant database like Cotality accurately price these homes in real time?

Saima Jewett: I think that's an ongoing, evolving issue right now as we speak. You know, public record data typically does not have that robust information, but our model, at the same given time right now, does not gather those data points. The model is not a boots-on-the-ground solution; therefore, it is much harder to collect that smart feature data. Now, the model does look at the likewise kind properties and provide a high or low range, acknowledging that it cannot account for every feature in the home without a physical inspection. I think for situations like that, a full appraisal is definitely needed so that you have your boots-on-the-ground appraiser who's physically walking not only on the exterior of the property but also the interior of the property. I think this can cross over not just from an automated valuation use case standpoint, but across the board, even from an insurance point, you know? We have all these companies that we potentially have a partnership with that gather that data from a smart feature standpoint, and some companies are doing that. Our model is just not there yet, and that is something that we are definitely looking into to accommodate in the near future here.

Allie Barefoot: Absolutely. And the new AVM rule requires valuations to be auditable, which, again, can be translated to mean transparent. And that's what a lot of home buyers are searching for. You know, once again I bring up that survey that shows that home buyers, still overall, they want a human explanation, which is great that you say that sometimes you need a full appraisal to really depict what your home entails. And there are so many layers of data and people behind these valuations. How do we know who or what is responsible for the results?

Saima Jewett: Yeah, that's actually—that's actually a really good question as well. You know, we work with over 3,100 different county assessors and recorders' offices to ensure that we have the latest public record data that is, you know, updated and available. We also do supplement that data with appraisal and listing data and understand the market. Responsibility of measure against the real results is how the house has sold on the market yesterday, and what do those results look like tomorrow? I think it is to have those responsible results and that validation is extremely important. So, we have this ongoing relationship with these assessors, and if something goes sideways, you know, we are pretty much up in par with them to make sure that the data that we're getting is correct. So, we're doing our job for sure as an industry leader to make sure that we drive these initiatives and we infuse the model with the best data as possible.

Allie Barefoot: And if a valuation model decides that a home in one metro area carries a high risk and a lower value, the buyers, insurers, and investors interacting with that data unknowingly absorb that bias. Can that bias be written into policy? How does the new AVM ruling help alleviate adverse consequences because of a bias?

Saima Jewett: Yeah, so before this ruling, the lender could simply say, "Hey, the model is mathematically accurate, right? It predicts the price correctly." But the new rule adds a critical, you know, component—a second test that requires us to test for non-discrimination specifically. It forces us to ask, "Okay, is the model accurate, but does it perform differently in majority of the minority neighborhoods compared to majority of the white ones?" If the error rate is higher in one zip code versus the other, we must now take a step back and say, "Hey, we need to look into this and we need to correct it." But in that area, if it performs worse than the other, it is normally because we have less amount of data in that area. We can't easily fill in the missing data, so there are ongoing data issues that we will continue to work on and address in those particular areas. But this is an evolving subject, and we are working very closely with a lot of the lenders to kind of solve for this. But we are very confident in the decisions and what we presently have from a model perspective as an industry leader, that we'reconfident in what we are presenting out in the world.

Allie Barefoot: That makes total sense, and that's definitely something we can continue to monitor looking forward and obviously have more and more conversations about the monthly Cotality Home Price Index report. Saima, thank you so much again for joining us here, and I look forward to many more conversations as the year goes on.

Saima Jewett: Thank you, Allie. Thanks again for having me, and I look forward to more conversations, too.

Allie Barefoot: And thank you again to Saima Jewett for joining me on Data in Context, and thank you guys so much for listening. If you haven't already, go on ahead and subscribe to the channel, and if you want to find out more information, as always, head over to cotality.com.