The Year to Adopt CATS

Part1: Why we need Common Ad Transport Standard (CATS), the privacy-first, extensible, and interoperable solution to transport information with a VAST ad request that is available now.

It’s time to have a chat about a temporary measure that was put in place to deliver information via macros to digital video ecosystem players. This core piece of technology is used to transport information via the Video Ad Serving Template (VAST) standard and is running into limitations that create technical challenges.

So, what are the technical challenges we collectively need to address as an industry? As more data protection regulations hit the books, privacy has taken the spotlight in digital advertising. Growing reliance on macros in the video ad ecosystem is the root of the challenge we’re seeking to face with new, industry-designed solutions.

When it comes to privacy signaling in video, as macros with encoded transparency and control messages roll out across the ecosystem, we are running into URL length limits, things are breaking, and data is not always being passed efficiently. If the industry doesn’t find a more sustainable and streamlined alternative, the problems will continue to grow — negatively impacting privacy signaling.

The underlying issues with current macro-based practices are simple to understand and the first step is to change the VAST ad request from a GET HTTP method to a POST HTTP method. Let’s go into more depth about the challenges that exist and discuss how CATS solves them.

The Problem with Macros

Let’s say we’re a party receiving privacy signals or other important information as part of a VAST request. We could be an advertiser ad server, a publisher ad server, a verification wrapper, or any other endpoint that speaks VAST.

When crafting our VAST tag URLs and handing them off to third parties for consumption, we’ll include macros in them. That is, for every piece of information that we want to receive from the third party, we’ll include a placeholder (the macro) which the party should replace with that information.

For example, say we have an ad server at ads.com that needs to know a few things about the context in which the ad is being viewed:

the value of the “TC String”, which indicates transparency and control settings;
the URL to the page where the ad is being inserted;
the dimensions of the player on that page.

The IAB Tech Lab and local policies define macros for all of these items (and many more) as part of the VAST specification, so we can request them from the party consuming our VAST tag URL for campaign number 42:

https://ads.com/vast/42?consent=${GDPR_CONSENT_1234}&page=[PAGEURL]&size=[PLAYERSIZE]

Apart from being a little esoteric, this gets the job done: the parts in bold will get recognized by third parties who have also implemented the VAST specification, they will get replaced with the right values, and our server at ads.com can consume those values.

So what’s wrong with this picture? Let’s highlight a few challenges:

Long Tag URLs

The example above contains just three macros, but usually, there’s a lot more that we want to know about the context in which our ad is being run. It’s not uncommon for tag URLs to contain tens of macros — granted, that doesn’t sound like a big number, but just try and imagine the tag above with 50 macros in the ad request URL.

Why is this a problem? Well, everything has its limits. The tag needs to go somewhere, and that receiver needs to be willing and able to accept a long URL. Notably, Google’s Display & Video 360 has a hard limit of 1,024 characters, and they’re certainly not alone in this. Different parties use different thresholds, and ideally, all tags should respect the lower bound.

However, at best, that practical limit is at a few hundred characters. How will you ever fit all your macros in there? And what happens when new macros are added to the specification?

Long Tag URLs (Wait, Again?)

Didn’t we just talk about URL length? Technically, not really! Actually, a VAST tag URL is more of a URL template, and the actual URL to request is formed by replacing the macro placeholders.

Let’s circle back to our example:

The [PLAYERSIZE] macro will get replaced by something like 640,360. That’s fewer characters than the placeholder, which is great!

Unfortunately, this is not always the case. The [PAGEURL] macro, for example, will get replaced with the full URL to the publisher page. Let’s say that URL is https://dev.iabtechlab.com/blog/vast-5-now-available/. That publisher URL will go inside our tag — cue Inception reference.

Finally, there’s our TC String macro ${GDPR_CONSENT_1234}. First off, the 1234 in there is an example of a vendor ID from IAB Europe’s Transparency & Consent Framework v2.0; feel free to disregard it. The macro’s value is a base64-encoded binary document containing a lot of important information. Here’s a real-world example:

CPBF8qoPBF8qoAGABCNLBLCoAPLAAAAAABpYGptX_T7dbWNC2f59ZtswOYxf9tCNJ-QjAAaJI2gBwRqQMBQGkmAanATgBAACKAYAKCJBAAJkGAAACQAQ4AAAAACASACABAIIICIAgAIRCAAIAAQCAIAARAAIgEACMEAAmwgAAIYgSCAAhAAggAAALEQCQAVABcAEMANQA6oCLwFIgLkAZOEgUgAIAAWABUADIAHAAPAAgABEACoAGgAPIAhgCKAEwAJ4AVQAsABcADeAHMAQgAhoBEAESAI4AS4AmgBSgDDgGoAaoA7wB7AD9AI4ASkAxQCLwExAKRAXIAvMBkgDJwgAmABwAHgAfAD-AL4AZoA6gDqgI9AamGgFgAqAC4AIYAakBaAFpAOqAi8BSIC5AGMAMnDAAgDqAL6HQNQAFgAVAAyABwAEAAIgAVAAxABoADwAH0AQwBFACYAE8AKoAWAAuABfADEAGYAN4AcwBCACGgEQARIAjoBLgEwAJoAUYApQBYgDDAGUANEAagA7wB7AD9AIsARwAlMBaAFpAMVAdMB1AEXgJBAVYAtkBcgC8wGMAMkAZOOANgAIgAcAB4AFwAPgA5AB-AF0AP4AvgBmgDqAHcAQgAiIBGQC2gF1gMAAwIBrwDpAHVAPIAj0BMQC-gGmgNTJQHwAEAALAAyABwAEQAMQAeABEACYAFUALgAXwAxABmADaAIQAQ0AiACJAEcAKMAUoAygBqgDvAI5AWgBaQDFAHUAReAvMBk5IAWAA4AC4AOQBfADUAHcARkAuoBrwDqgL6KQKQAFgAVAAyABwAEAAIgAVAAxABoADyAIYAigBMACeAFIAKoAWAAuABfADEAGYAOYAhABDQCIAIkAUYApQBYgDKAGiANUAd4A_QCLAEcAJSAi8BcgC8wGMAMkAZOUALAAXAA-ADkAH4AbQBHAC-AGoANcAdQA7gC6gGAAMUAa8A6oB5AEegJiAX0A00BqYAAA.YAAAAAAAAAAA

We now have all our macro values, so let’s finally build that ad URL!

https://ads.com/vast/42?consent=CPBF8qoPBF8qoAGABCNLBLCoAPLAAAAAABpYGptX_T7dbWNC2f59ZtswOYxf9tCNJ-QjAAaJI2gBwRqQMBQGkmAanATgBAACKAYAKCJBAAJkGAAACQAQ4AAAAACASACABAIIICIAgAIRCAAIAAQCAIAARAAIgEACMEAAmwgAAIYgSCAAhAAggAAALEQCQAVABcAEMANQA6oCLwFIgLkAZOEgUgAIAAWABUADIAHAAPAAgABEACoAGgAPIAhgCKAEwAJ4AVQAsABcADeAHMAQgAhoBEAESAI4AS4AmgBSgDDgGoAaoA7wB7AD9AI4ASkAxQCLwExAKRAXIAvMBkgDJwgAmABwAHgAfAD-AL4AZoA6gDqgI9AamGgFgAqAC4AIYAakBaAFpAOqAi8BSIC5AGMAMnDAAgDqAL6HQNQAFgAVAAyABwAEAAIgAVAAxABoADwAH0AQwBFACYAE8AKoAWAAuABfADEAGYAN4AcwBCACGgEQARIAjoBLgEwAJoAUYApQBYgDDAGUANEAagA7wB7AD9AIsARwAlMBaAFpAMVAdMB1AEXgJBAVYAtkBcgC8wGMAMkAZOOANgAIgAcAB4AFwAPgA5AB-AF0AP4AvgBmgDqAHcAQgAiIBGQC2gF1gMAAwIBrwDpAHVAPIAj0BMQC-gGmgNTJQHwAEAALAAyABwAEQAMQAeABEACYAFUALgAXwAxABmADaAIQAQ0AiACJAEcAKMAUoAygBqgDvAI5AWgBaQDFAHUAReAvMBk5IAWAA4AC4AOQBfADUAHcARkAuoBrwDqgL6KQKQAFgAVAAyABwAEAAIgAVAAxABoADyAIYAigBMACeAFIAKoAWAAuABfADEAGYAOYAhABDQCIAIkAUYApQBYgDKAGiANUAd4A_QCLAEcAJSAi8BcgC8wGMAMkAZOUALAAXAA-ADkAH4AbQBHAC-AGoANcAdQA7gC6gGAAMUAa8A6oB5AEegJiAX0A00BqYAAA.YAAAAAAAAAAA&page=https://dev.iabtechlab.com/blog/vast-5-now-available/&size=640,360

Yikes! That’s one long URL. And in reality, it’d be even longer than that: the VAST specification mandates that macro values be URI-encoded: in order to avoid conflicts, and to respect HTTP in general, special characters must be replaced with escape sequences that start with a percent sign. We left those unencoded in our example for the sake of readability, but in the actual URL, you would see a lot of percent signs.

This was just a toy example rather than a production tag containing a bunch of macros, and already, the URL’s length exploded. However, we talked about macro values rather than the tag template. Therefore, none of this is subject to the tag length limitations above, so why should we care? Well, the long URL with all the macros replaced is the one that the client (usually a browser) needs to go out and fetch using HTTP.

How long a URL does the HTTP standard allow? The answer is “it depends.” Technically, it’s less than 2,000 characters, but in practice, most servers (and intermediaries such as proxies) support more than that. And if the URL is too long, will it get truncated or will the request fail altogether? Again, it depends on the implementation. Do you really want to take that risk?

Reserved Characters

Looking at our example tag URL, there’s clearly an entire arsenal of special characters at our disposal. Some stem from HTTP, some are part of macros.

Our example only uses IAB-standard macros. In reality, most platforms extend this suite with their own custom macro format, which allows capturing platform-specific information. It’s not uncommon for tag URLs to also contain macros that use, for example, %% as the delimiter.

However, when you pass a tag URL off to a third party, that party will usually validate that it’s properly formatted. That’s a noble intent, but what if that party happens to be unaware of the fact that %%MACRO%% will actually get replaced at some point? Without that key piece of information, it will just see two consecutive percent signs, which is invalid in a URL (and not a VAST URL template), and the tag may very well get rejected.

In particular, this poses another challenge for the TC String in IAB Europe’s Transparency & Consent Framework. TCF macros are defined there rather than by VAST itself. Their delimiters aren’t VAST’s square brackets, but instead, ${ and }. You’ve probably guessed it: some systems may not accept those.

Vendor Interoperability

Many parties tend to be involved in the orchestration of a video ad. Those parties may or may not be interested in the same pieces of information.

The standard way of chaining multiple vendors together is to use wrappers. Many variations exist, but they all involve some way of having one tag reference another, including its macros. In such situations, each vendor in the chain is responsible for opting into specific macros, often leading to a lot of repetition.

Additionally, in more complex wrapping scenarios, each party doing wrapping is put in charge of relaying macro values between their tag and the wrapped one. Because there’s a linear chain from one vendor to the next, information can easily get lost, as no vendor implements every single macro out there. This means that macro values may unintendedly get dropped along the way.

As an example, let’s say we’re a party that wants to add some basic tracking to existing tags at scale — a primitive verification vendor, if you will. VAST wrappers exist for this purpose, so said party spins up a Web service that takes in an existing tag via the URL, generates a VAST XML <Wrapper> around it, and sends it back to the user. Their URLs might look something like this:

https://extra-trackers.net/wrapper?wrapped=https://ads.com/vast/42?consent=${GDPR_CONSENT_1234}&page=[PAGEURL]&size=[PLAYERSIZE]

Wait, that can’t be right. The value of the wrapped parameter contains special characters, so how does the server know that the ampersands in there aren’t separators for extra-trackers.net? Yet again, URI encoding to the rescue: every special character must be replaced with a percent-sign escape sequence.

https://extra-trackers.net/wrapper?wrapped=https%3A%2F%2Fads.com%2Fvast%2F42%3Fconsent%3D%24%7BGDPR_CONSENT_1234%7D%26page%3D%5BPAGEURL%5D%26size%3D%5BPLAYERSIZE%5D

Wow, what a mess — now you know why we left out the escape sequences earlier. But… where have our macros gone? All the special characters used to delimit macros were also percent-encoded. As a result, the party consuming the tag will no longer see them, and extra-trackers.net won’t receive their values. Yep, we’ve broken our tag.

To work around this, should we revert encoding for the macros only? We could, but that would just shift the problem. After the [PAGEURL] placeholder has been replaced with an actual URL — which is URL-encoded a single time per the VAST specification — how will we distinguish it from the rest of the URL? Indeed, this is a tough nut to crack, to the extent that the video team at DoubleVerify actually uses it as a question on job interviews!

People Make Mistakes

By now, you’ve probably noticed that while macros are a seemingly simple and elegant solution, they’re actually fairly complicated. Inevitably, all this complexity leads to mishaps.

Those mishaps can be as simple as typos, but also misinterpretations of specifications, general URL encoding issues, edge cases that weren’t considered, and so forth. The result? A loss of information, and in the case of privacy frameworks, perhaps even a violation of the user’s rights.

Conclusion

As you can see, the temporary measure to deliver information via macros to ecosystem players is starting to run into challenges. Character limits leading to important details being omitted during an ad call, reserved characters causing requests to fail, and macros are prone to human error. It’s possible that VAST in its current incarnation comes with more challenges than you thought. That’s okay though, because there is already a solution.

Enter CATS, the Common Ad Transport Standard. CATS turns the HTTP GET request for a VAST tag into a POST, allowing clients to relay a rich set of metadata. Just like that, macros and their associated challenges are a thing of the past. Boom!

If it will solve all of our current challenges with macros, why isn’t everybody using CATS yet? Well, it’s not that simple. In our next post, we’ll take a closer look at how CATS solves all of the challenges we’ve outlined above, what the road toward broad adoption in the industry looks like, and what the future holds after that.

The IAB Tech Lab would like to help put together a set of vendors who would like to address the issue and can work together to send and receive this information for testing. If you would like to help be an agent of change with this effort, please reach out to the Tech Lab.