Tealium iQ Version Control

[IgorPluznikov]

Hi all, 

We currently have 5 people working in our Tealium iQ setup. Therefore, we are looking to improve out branching / versioning strategy.

"Merging Versions" feature seemed really nice. We decided to create a "master version", similar to a e.g. "master" branch in github :) And then we would create a separate version for each feature that we develop in Tealium. Once a feature is production ready, we would merge that feature version into "master version" and then release "master version" in the end of a sprint. However, our idea fall apart when we noticed that "Merging Versions" is not supporting Profile Inheritance (Libraris), that we heavily utilize.

Does anyone has best practices to share when it come to versioning / release strategy covering a fairly big team working in the same Tealium setup?

[UnknownJ]

Yeah, like, this used to deeply upset me that I couldn't do cool branching stuff while still supporting library inheritance, but in a sense I think it sends the right message, which is that library inheritance sits in complete opposition to meaningful version control.

I'm not saying it would be impossible to blend the two things together, but fundamentally when you work with library inheritance you give up the ability to roll back changes in a straightforward fashion because the changes in question might sit within one or more inherited libraries, and rollback involves going into each, rolling those back, and then going back and republishing your main profile, and it's not that you can't do that, you just can't do it while holding in your head the belief that you're doing effective version control.

Essentially, what it then boils down to is how to adopt one pattern whilst retaining as many of the benefits of the other as possible. In my case, that's been to keep library inheritance but to offload more of the TIQ work to JS extensions whose code can be managed in Git and either pasted, loaded via the UI integration with Github, or via the new profile update APIs. It doesn't give you the ability to encapsulate interface-level config (such as tags and load rules) into a branch with your code, but if the bulk of the complexity of a large deliverable sits within JS extensions, it gets you at least part-way there to version control and delivery of completed features into a main branch.

The inverse then of that is to run with monolithic profiles and to find ways of separating and potentially reusing components using methods other than library inheritance. There's a sort-of-undocumented Automator tool that can do some of the heavy lifting if you needed a given component to be present in multiple published profiles that would allow you to master it in one place and programmatically copy it to others, which at least starts to solve for the "build once, use in many places" feature of library inheritance. The user access rights angle is harder to replicate - in my head, I've got this idea that you could hypothetically retain child libraries with specific access rights, and then use the various APIs and tools available to take snapshots of those libraries and insert them into a main profile that's not linked to them within TIQ but which would effectively inherit those configurations via that copying process. I could imagine that adding a whole heap of tech debt to your implementation, creating awkward dependencies on non-standard automations that make it harder for Tealium (and colleagues) to support your setup, and creating some potentially uncomfortable dependencies on very specific API methods that aren't guaranteed to work forever.

So like, no easy answers. For my part, I really like turning everything into pure JS and then mastering it via Git, and treating version control and branching as something that's done upstream of TIQ itself. But if you use a lot of the interface-led extensions or need to bundle tags, load rules, UDO elements and extensions into overarching branches, it's not going to work out so well..

[IgorPluznikov]

Hi UnknownJ,

Thanks for your great reply, it is exprimily helpful for me since I am new to TIQ.
All the options that you have mentioned are essentialy good in their own ways, but might not produce the desired outcome for all types of teams though, but that is also what you say in the end of your comment.

I personally very much like the github/JS/Extension approach. However, one of our key objectives is to simplify our TIQ setup, and one of the ways to do it in our case is to reduce JS complexity, so that less tech savy team members and stakeholders can also deliver in an efficient way. And for the same reason, I believe that creating a custom sollution for library inheritance might not solve our core challenges. 

We are currently working on consolidating out multi-profile setup into just few profiles. This might be a good opportunity for us to re-think our libraries and the need for them. But that would be a tough decision, since libraries have quite few advantages such as separation of concerns, inheritance etc. 

[dtawil]

@IgorPluznikov @UnknownJ This is a great discussion and one that is of particular interest to the team here as we have been discussing ways to imrpvoe the versioning process, specifically as it relates to libraries. 

Like you mentioned, being able to merge libraries into a profile in the same way you can merge two profile versions does become problematic. The library should be used to ensure the consistency of configurations across multiple profiles. Having the ability to granuarly pick and choose the elements from a library to merge into the profile comes with it's own set of versioning challenges and also somewhat defeats the purpose of the library in the first place. 

From our persective it seems like there are 2 potential issues at play here that prevent better version control when using libraries, but I'd be curious to hear your perspectives on the matter and see if there is something we are missing, oversimplifying or missunderstanding.

The first issue is that once a library gets inherited it is forever that library version in the profile, rendering a roll back to a truly accurate state nearly impossible. Referencing the flow diagram below, in today's state if I were to roll back from V3 to V2 I'd still see V2 of the library which gives me an innacurate view of what V2 actually was. I could go back into the library and roll back that version as well, but then this throws other unimpacted profiles out of whack and just complicates things more than need be.

But what if a library version was fixed to a profile version? For example (reference the diagram) what if: 

• V1 of the library is fixed to profile V1.4, V2, V2.1, V2.2
• V2 of the library is fixed to V2.3, V3 of the profile

This way if I were to roll back from V3 to V2 I would see V1 of my library, thus preserving the sate of my profile versions. This would of course require you to re-inherit V2 of the library when ready, but would at least keep a more preserved historical state. 

The second issue is that you can't merge profile versions when that profile inherits from a parent library. What if that restriction were removed and now V2.1 and V2.2 could be merged together? The library version is the same so no harm no foul, you just merge the profile changes. But this doesn't cover all usecases, what If I wanted to merge V2.2 and V2.3? The libraries are different. If you had the ability to merge the library version as part of the merge process, so that V2.2 would contain the full library V1 and V2.3 would contain the full library v2 this would give you the option to pull V1 or V2 of the library forward into the next profile version. 

What are your thoughts? 

Regards,

Dave Tawil | TiQ Product Manager

[UnknownJ]

In terms of rollback in a profile that has library inheritance, I guess my issue is that it does not technically exist. You can fix-forward by reverting to older snapshots of various libraries / the profile itself but the concept of a true rollback is not one that's been implemented, but presumably could be..?

Each publish produces a distro bundle, to enact a genuine rollback you would simply need to push an older distro to the CDN, perhaps sensibly constrained such that only a previous prod build could be pushed to prod.

Then for the more granular stuff, when to inherit etc., for me I think it makes the most sense to have some sort of timeline tool not unlike the current client-side versions feature, which in a profile would give you the ability to look across all the inherited libraries, and using a master slider pick a point in time at which to inherit the latest (at that point) version that's eligible for inheritance. And then beneath that, library-level sliders that would allow the user to set custom inheritance on a specific library if they so wished.

The default would always be "current", and maybe the standard behaviour would be that the slider snaps to profile publish events, so the simple-mode experience is that you can simply drag the whole set of inherited libraries to their state at a given publish within your profile. But the complex-mode experience would let you pick any moment in time, and split those timings by library.

Use case would be - three days ago I performed a production publish and one of the tags that's been configured in an inherited library is leaking a data item to a vendor that they should not be receiving, and it's severe enough that I want to be able to perform a fix without reverting the entire publish. But time has passed, new features have dropped in other inherited libraries ready for the next scheduled publish, but I need to perform a hotfix now to sort this one issue and I'm not intending to launch any new features. The owners of the affected library have ensured that what's published there today is what was in the last release, plus the fix, nothing else, but I can't go around another five teams' libraries reverting everything they've done this week, perhaps I shouldn't even have the access to do so. So, I go into the Timeline view in simple-mode, drag the master slider back to our last production publish, enable complex-mode, and then find the affected library and drag its slider to "current". Now the content of my publish is the inherited libraries at their position at my last publish except for the one library that requires an update, which is inherited in its current state.

The way I'm envisaging it working in the backend would be effectively that each library publish is in effect a commit to the branches (environments) to which it's published. From my profile I don't see details of the development branches that users within the library are using to manage their own code, but I do see a stream of commit/publish events that I can roll back to a specific point in time for the purposes of what I'm inheriting.

I'm probably just reinventing Git submodules or something, so this might all be terribly derivative, but you get the idea..

Bonus feature: The ability to specify for a given publish what the minimum version of other libraries/profiles within the account needs to be for that publish to work. Because interdependencies between inherited libraries do exist and they're awkward. If I make a change to my Webchat library to enable a new supplier and my Analytics library contains code that adds event listeners to the new webchat dialog, I need to be able to specify within my Analytics library that a given publish expects that the Webchat library will be at least as recent as insert date here. Or use tags as sort of feature flags so that you have a dependency on a specific tag being present in the other library, I dunno.. but what you want to avoid is a scenario in which an interdependency between two libraries is a) not made visible to users and could be forgotten about and b) cannot cause weird side-effects if one library is rolled back and another that depends on now-reverted code isn't rolled back.

And finally, the feature I'm describing above would also work for maintaining stable releases vs nightly release tracks etc., wherein you might have a main profile that takes the most recent version of every library it inherits, and a near-identical main-stable profile that has all its sliders fixed at a given point in time, only moved when a new version is considered stable having been shown to be issue-free in production. "Can't you just do that already, freezing the stable version by not publishing the profile?" you might ask, and not quite, because at the point of publishing that profile, you're getting all the absolute latest versions of the libraries, which might not be the versions that you want to include in your stable release. Furthermore, if you needed to make any reference data changes (e.g. update a marketing tag campaign ID that has expired) without functional code changes, you'd have to re-inherit the whole set of libraries at their current state including all code.