Applying Tech and Data to Reimagine Vendor Evaluation
Overview
More than just your standard digital publisher, Assembly is an award-winning company providing next-level technology-powered digital publishing. Alongside Assembly’s Director of Technology sits Andrew Fowler, Director of Ad Operations. Unlike other AdOps Directors, Andrew has a strong background in engineering, with roots in software development and computer science.
As Andrew explained, “We’ve gone about building our publishing business from a tech company perspective.” When Assembly’s contract with GeoEdge was nearing its end, the team saw fit to put GeoEdge and other players to the test. Statera- Assembly’s proprietary vendor evaluation tech, offered a perfect opportunity to identify the most effective ad quality and ad security vendor on the market.
Problem
While Assembly is more software development-forward than traditional digital publishers, Andrew and his team faced the same questions as all digital publishers: How do you evaluate and choose the right vendor?
Even with great partners on your side, the end of a contract is an opportunity to consider other vendors. Assembly’s proprietary internal analytics platform named Statera is used across the board to inform business decisions, including selecting vendors. Statera is so ground-breaking that it earned Assembly the 2019 Digiday Award for Best Use of Audience Insights.
Statera analyzes billions of data points, identifying shared characteristics layered over this platform is Assembly’s multi-variant testing software suite, which empowered them to evaluate the performance of different vendors.
As Andrew explained, “If we’re trying to add a new advertiser into our programmatic advertising stack, web core vitals (LCP, FID, CLS) is huge for us.” What truly makes Assembly’s “tech company approach” stand out is that they own their data.
The Solution
Statera offered Assembly a perfect opportunity to identify the most effective ad quality and ad security vendor on the market. “We haven’t always had Stratera,” explained Andrew. Over time, and with rigorous testing, Statera had become more polished.
Andrew and his team narrowed down their list to three potential vendor candidates based on the key features they were looking for – specifically, creative review, blocking controls, landing page scanning, and IAB category-based blocking. It was then time to put what they had built to the test.
Assembly ran a multivariate test with three different ad quality vendors. In the performance test, Assembly served each one of their three potential ad quality partners one-third of their US mobile traffic.
“We had a few metrics that we were measuring them on,” Andrew noted. “We can see how each vendor interacts and impacts core web vitals, how they impact ad load speed and in turn ad viewability, and we measure them all against each other.”
Not every metric was as easy to measure, however. For example, which of the three vendor candidates was the best at blocking ads? “That’s a tough one,” Andrew admitted. “One (reason) is because the settings for all the ad quality vendors are a little bit different on their dashboards. In order to run a fair test, we tried to keep all the settings the same across all three contenders, but I must admit it wasn’t perfect. We couldn’t have it exactly the same, but I think we got pretty close.”
Outcome
Assembly ran its evaluation tests for about a month, and the results showed that GeoEdge beat out the other two vendors. GeoEdge had the highest number of unwanted ads blocked along with the lowest impact on ad load speed. That last metric is the one that really swayed the odds in GeoEdge’s favor, since all three competitors were comparable across categories like core web vitals, script size, and impact on the page.
Fowler explained that when dealing with data and software development, it can sometimes be very cold and calculating. In reality, however, publisher relationships with vendors are based on a bond of trust and connection.
Fowler knows first-hand how easy it is for publishers to get stuck in a trap of constantly cycling through vendors every time a contract expires. “Maybe it’s because of pricing, or a better sales pitch,” he shared. But somehow, too many of us end up on this treadmill, “going around and around and around, trying out different vendors and not knowing if they’re improving or degrading.”
