This paper is designed to help educate Tealium users before implementing an A/B testing tool. It will not explain how to implement any tools, but rather what considerations should be made before implementation begins. There are other vendor specific posts that will explain the implementation process.
Before we get into the weeds it is good to note that Tealium supports both synchronous and asynchronous methods of implementing A/B testing tools. The method of implementation should be dictated by:
Since Tealium is vendor neutral, most of the time the testing vendor will dictate the method used for implementation.
It is also good to note that using an asynchronous method does not always guarantee problematic flicker, and, likewise, using synchronous does not always guarantee there won’t be flicker. It is very dependent on your specific site, your specific requirements and where on the page the Tealium utag.js file is loading. However, flicker is likely with an asynchronous implementation.
First let’s explore the differences between the way things are loaded synchronously and asynchronously.
This is the legacy method of doing tag deployment via Tealium. The industry is moving away from this method because it presents several challenges. Most notably, Google transitioned away from synchronous loading of its popular Google Analytics product because of the added risk and performance degradation associated with this method.
Synchronous tags load in a serial fashion. When a synchronous tag starts loading, it blocks the page from loading. Once the tag loads, then the rest of the page can continue to load. This is analogous to a slowmoving truck in a onelane road that will slow down the entire traffic behind it.
One benefit of this model, however, is that it can effectively control the loading of the rest of the page. This comes in handy when the goal is to eliminate what’s known as “flicker.” Flicker happens when the default content loads, followed by the test content in front of the user. For example as the page loads, the user may see a green calltoaction button and a split second later it is replaced with a red button.
The downside associated with this method of course is that the entire site is blocked from loading until the tag fully loads. Although vendors put in place SLAs around their delivery time, several factors could impact their performance.These include slow response time associated with vendors, introduction of unnecessary application servers in the ClientServer hybrid model, slow Internet traffic, etc. If you are loading tags synchronously, we recommend that you make sure the vendor has a response time of 100 milliseconds (ms) or faster.
The more modern method being adopted by the industry is to load tags asynchronously. In this method, tags are loaded in parallel to the rest of the page content. This means that even if the tag is slow to respond or to load, it will not slow down the rest of the site. This is demonstrated in the figure below.
Although this method is preferred for most tags, it can present some challenges for testing tags in that it can potentially introduce flicker in the user experience if either the TMS or the testing servers are slow to respond. The flicker of course may or may not be an issue. For example, if your tests present different prices, then flicker is an important consideration. However, if the tests are around other pieces of content (i.e. call to actions, color or design of offers, etc.), then flicker can be easily tolerated. Again, the magnitude of the flicker will be dependent on the response time of the TMS. By selecting a vendor that has a response time of 100 ms or faster, you ensure that the potential impact of flicker is negligible
To summarize, one of the main decisions when implementing testing tools is whether to deploy the vendor synchronously or asynchronously. Each method presents a series of pros and cons. With synchronous deployments, organizations can almost always guarantee that their tests are presented in a flickerfree fashion. Unfortunately, this could come at the cost of slowing down the entire site or loading a blank page, which could be far more costly than presenting a small amount of flicker.
By deploying the testing vendor asynchronously, companies can ensure that their site will not take a performance hit, but could potentially experience flicker if a vendor’s servers are slow to respond. Also, by following Tealium best practices when implementing asynchronously you may be able to remove flicker completely or cut it down significantly.
Regardless of implementation method, the Tealium utag.js file will be cached after the visitor’s first page view, any Tag will load and fire very quickly on any subsequent pages. This is very beneficial when working with asynchronous implementations. The first page view is typically the slowest page to load the Tealium tags and testing code. However, thanks to the benefits from gzipping and the Tealium feature of bundling, the first page loads pretty quickly as well. Additionally, the Tealium “MultiCDN” and gzipping features will not only load, but also run the necessary libraries very quickly. It's really the content on the site that will be the biggest factor dealing with flicker.
Follow New Industry Standards - Some clients want to follow the industry and do not want to load anything on their site synchronously. In this case, one of Tealium’s asynchronous methods should be investigated.
Risk Factor - Is the risk of a synchronous script worth the potential flicker? As pointed out throughout the document, some flicker is very inconsequential in that it will have little impact on user experience. Additionally, using asynchronous does not guarantee problematic flicker.
Test traffic - What percentage of your traffic will be exposed to tests? If it’s a low percentage then the asynchronous method is the best option. Say for example that your site gets 1 million visitors per month and you’re running tests on cart and checkout pages that only get 5% of the site traffic. Then we don’t recommend putting your site performance at risk for your entire user base in order to avoid flicker for 5% of your audience.
The true effect of flicker - If your tests are around design and calltoaction messaging, then the flicker effect is not as critical. For example, a visitor is presented a red button, and a splitsecond later sees a green button. Also, if the test happens below the fold then flicker won’t be noticed. The primary area of concern for flicker should be around change in price or offer. For example, a visitor may see a 10% discount offer, and a split second later sees a 20% discount offer.
Location of test - The other factor to consider is which pages will be tested. If tests are done on internal pages (i.e. article pages, product pages, etc.), then the chance of flicker is almost zero since most files will be prefetched and cached. The biggest chance for flicker to occur typically happens on landing pages and therefore your choice should take this into account.