Tealium Learning Center

This document describes optional review steps and how to initiate the first training for your model: In this document: Table of Contents Placeholder Review your Model Before Publish (Optional) Use the following sections as a guide to review your model and start training. Revising your model using these steps is optional. If no changes are needed, skip to Save and Publish to Start Training. Add Labels (Optional) Add a label: To add a label to improve search results, click on a model to expand the details. Click the Add Label drop-down list and use the checkboxes to select one or more labels. Click Apply. Edit the Model Name (Optional) View and edit the your model name: Click on a model to expand the details. Click Edit. The Edit Model dialog displays. Edit the model name and descriptive notes. Click Save. View or Edit your Target Attribute (Optional) To view or edit your target attribute: Click on a model to expand the details. Click the drop-down menu to the right of the Edit button and select View Target Attribute. The details for your target attribute display. Click the pencil icon to modify the attribute. The Edit Attribute dialog displays. Make the desired changes and then click Save. View or Edit your Output Attribute (Optional) To view or edit the output attribute for your model: Click on a model to expand the details. Click the drop-down menu to the right of the Edit button and select View Output Attribute. The details for your output attribute display. Click the pencil icon to modify the attribute. The Edit Attribute dialog displays. Make the desired changes and then click Save. Save and Publish to Start Training Your model is currently in the Requires Publish stage. Use the following steps to start training: Save and Publish your model. Tealium recommends using the Save As option and selecting a meaningful title and description. Before saving and publishing, your model has been saved, but not published. If no errors are encountered, training begins automatically upon publish. Click OK. If no errors are encountered, your model is now in training. Your model is now Version 1 with a status of In Training. Training is a one-time event and each retraining results in a new version number. For example, when you retrain Version 1, the resulting version is Version 2. Your model is now in training. Check to ensure the model status now displays In Training. Once the training date range has lapsed, the model status is updated to Requires Publish to Deploy. Next Steps For next steps, go to Evaluating the Trained Version. ... View more

This document describes how to use optional advanced configuration options to select a training date range and select attributes to exclude, if desired. In this document: Table of Contents Placeholder Configure Advanced Options Use the following sections as a guide to configure advance options for your model. Select the Training Date Range Select the training date range you want to use: Under Choose Training Date Range, select Use all available data or Custom. Use all available data is the default option and is recommended. If you select Custom, use the calendar drop-down menu s to select a date range. Selecting a prediction timeframe determines the date range for the data you want to train on. The default value for the second date range is today's date. The date you select must be in the past and cannot be greater than today's date. You can select dates as far back as you want from the point when your profile began collecting data. Choose Attributes to Exclude (Optional) Predict evaluates all attributes and selects the most predictive attributes by default. To improve accuracy, you can optionally choose to exclude one or more attributes from your model. Choose additional attributes to exclude: Under Choose Attributes to exclude, select Use all available attributes or Select attributes to exclude. Use all available attributes is the default option and is recommended. If you selected Select attributes to exclude, use the drop-down list to select one or more attributes to exclude from your model results. To exclude more attributes, click the plus sign (+) as many times as you need and select another attribute from the drop-down list for each exclusion. To remove an attribute you previously selected, click the minus (-) sign to the right of that attribute. Click Finish. The first time you add a model, you are prompted to accept the terms of use in order to proceed. Read the agreement and, if you accept, click I accept the terms of use. Your new model now displays as Version 1 with a status of Requires Publish. Model versions are automatically numbered consecutively, in the order they are created. Next Steps To continue, go to Review and Start Training. ... View more

This document describes how to add a model, select your target attributes and prediction timeframe, and then name your model and attribute for future use. In this document: Table of Contents Placeholder Get Started by Adding a Model Use the following sections as a guide to add a model and get started. Add a Model and Select your Target Attribute Add a model and select your target attribute: In the left sidebar, click Predict. The Predict Overview displays. Click + Add Model. The Add Model dialog displays. In the What would you like to predict? field, select a target attribute from the drop-down list. This is the AudienceStream attribute you will use to define your model and represent the user action being predicted. You can use the plain language indicators and health status that displays to the right of each attribute to help you decide. Select a Prediction Timeframe Select a prediction timeframe: In the What is your prediction timeframe? field, enter a number for the number of days, weeks, or months you want to predict. Select days, weeks, or months from the drop-down list. Name your Model and Output Attribute Name your model and output attribute: In the Name your Model field, enter a descriptive name for this model. Use a meaningful name, for example: Likelihood to Purchase in Next 1 Days. In the Output Attribute Name field, accept the default name based on your model name or enter your own descriptive name for this output attribute. In the optional Notes field, enter any descriptive notes that will help users understand what this model is intended to predict. Click Next to proceed to the next screen, where you will choose the data to act on. Next Steps To continue, go to Advanced Configuration Options. ... View more

This document describes how to delete a model. You cannot delete a version of a model, only an entire model. Use the following steps to delete a model: Go to the Model Explorer screen. Click the model you want to delete to expand the view. Click the menu to the right of the Edit button and select Delete. Click Yes to verify that you want to delete the model. The model is deleted and no longer available for configuration or training. ... View more

This document describes how to deploy or undeploy one or more versions of a trained model. In this article: Table of Contents Placeholder Final Steps to Deployment You are now ready to deploy your trained model. Use the steps in the following sections as a guideline to deploy one or more versions of a trained model and you are on your way. Select a Trained Model and Deploy Select and Deploy a Trained model: Click on the trained model and version you want to deploy to expand the details. From the Status column in the summary, click the Trained status. Click Deploy. Click Deploy again to confirm you want to deploy this version of this model. Your model is now Version 2 with a status of Requires Publish to Deploy. Save and Publish. Your model is now Version 2 with a status of Currently Deployed. Undeploy a Deployed Model To undeploy a deployed model: Click on the deployed model and version you want to undeploy to expand the details. From the Status column in the summary, click the Currently Deployedstatus. The training details for the version display. Click Undeploy. Your model is now Version 2 with a status of Requires Publish to Undeploy. Save and Publish your changes. After you save and publish, this status of this version of your model returns to Trained. Next Steps Repeat any steps in this documentation to add and deploy new models. ... View more

This document describes how to retrain a model after evaluating the model and determining changes are required to improve predictions. In this document: Table of Contents Placeholder Determine Needed Changes and Prepare for Retraining Use the following sections as a guide to prepare your model for retraining. Edit a Trained Model To review and edit a trained model: Click on a trained model to expand the details. Click the drop-down menu to the right of the Edit button and select View Target Attribute or View Output Attribute. See Review and Start Training for detailed steps. Review the attributes and click the pencil icon to adjust as needed. Click Save. Save and Publish your changes Retrain the Model To retrain your model: Click on the trained model to expand the details. From the Status column in the summary, click the Trained status. Training details for Version 1 display. Click Retrain Model. Adjust Advanced Options if needed, such as: Training Date Range Attributes to Include Attributes to Exclude Click Finish. Click Retrain Model again to verify that you want to proceed with retraining. Your model is now Version 2 with a status of Requires Publish. Save and Publish your changes. Your model is now Version 2 with a status of In Training. View Retrained Results for Deployment Readiness Once your new version is trained, see Evaluating the Trained Version to view the results of retraining. Continue to adjust if needed. When satisfied, proceed to deploy your model. Next Steps Once you are satisfied with your results and ready to deploy, go to Deploying a Model. ... View more

This document provides detailed descriptions of model strength quality scores. In this document: Table of Contents Placeholder Understanding the Quality of Your Model The quality of any machine learning model created using Tealium Predict or any other Machine Learning tool will vary depending on several factors, including (but not limited to) the following: The attributes which comprise the dataset and how complete of a picture the combined attributes provide/ The daily volume of visitors and visits in the dataset. Higher volume typically means there is more data available for the model to learn on. The Training Data Range selected for the model. Training a model for a longer time period generally equates to more data available for the model to learn on. In order to understand whether a given model is of high enough quality for use in the real world, machine learning experts typically use a combination of interrelated metrics. The specific combination varies depending on the type of model being evaluated. Tealium Predict enables non-experts to create models, evaluate the quality of the model, and take real-world actions in the most streamlined manner possible, while retaining with enough transparency for an expert to see the typical metrics they expect. The following sections provides explanations of the metrics and ratings available with Tealium Predict to help you understand the quality of your models. For a more comprehensive explanation of other scoring reports, such as the F1 Score, Confusion Matrix,  the ROC/AUC Curve, and Probability Distribution, see Evaluating Success. Accuracy In the context of Tealium Predict, accuracy is simply the ratio of correct predictions to the total number of predictions. Accuracy alone can give you a false sense of security and therefore cannot adequately judge the quality of a machine learning model. As an example, assume that you have red and green apples and 99% of the apples are green. In this case, our model could easily train to obtain a 99% accuracy score by simply predicting that every single apple is green, since that would be correct 99% of the time. This model, with 99% accuracy, would not be a good predictor of which apples are red. Model Strength Ratings The model strength ratings used in Tealium Predict provide an easy-to-understand rating of the quality (strength) of each version of each trained model. The rating system is comprised of four categorical labels. These labels are based on F1 Score, which is the typical metric used to evaluate the quality of a propensity model. F1 Score values are categorized using the following scale: Poor – F1 score is less than or equal to 0.50 Fair – F1 score is at greater than 0.50 but less than 0.60 Good – F1 score is greater than 0.60 but less than 0.80 Excellent – F1 score is greater than 0.80 The quality of any model is a relative judgment, not an absolute fact. Different teams have different needs and goals for their models, as well as different levels of sophistication in their modeling and testing abilities and varying quality in their input datasets. For these reasons, model strength ratings are not regarded as absolute. The intention is to use the ratings as a general guideline for quality. There are two types of strength ratings in Predict: static ratings for each trained version, and ongoing dynamic ratings for each deployed model. Strength Ratings for Trained Versions The strength rating for each trained version of each model is intended to provide an easy-to-understand rating of the quality (strength) of the training and the model which resulted from it. Models not yet deployed are not yet assigned a Strength Score. Training is a one-time event. Each retraining is a new, separate event, which makes this type of strength rating static and specific to each version. The rating for a version does not change over time. The strength rating is visible next to each version on the Model Explorer page, in the Training Details panel, and in the Tiles view on the Overview page (for the latest trained version). Strength Ratings for Deployed Models All machine learning models naturally degrade in quality over time, just as the real world continually changes. Models eventually stagnate and degrade in their ability to make accurate predictions about the changing environment. Models must therefore be retrained periodically to ensure that the model is based on the freshest information available. Live Model Performance Scores For deployed models, Tealium Predict recalculates the ongoing F1 score daily using the most recent time window available. This window is defined in the Prediction Timeframe, which is the number of days specified in the model as "in the next x days". In order for this nightly recalculation to begin, the model must be deployed for the initial Prediction Timeframe (30 days in this example) so that actual true/false results are known and can be used in the calculation. The calculation window then moves forward one day, every day. The deployed live performance score allows you to understand the ongoing quality of your model and know when it has degraded, which will inevitably happen over time. ... View more

Visitor data is not purged by default. For information about purging specific visitor data, contact your account representative. Visitor Data Usage Model training is not a one-time task. For that reason, visitor data is not purged. There will always be a need for new versions of visitor data to be trained and the data must therefore be available to retain. After a successful model is deployed, the model degrades over time and needs to be refreshed with current visitor data. Data Storage and Access Controls All data related to Tealium Predict remains within the Tealium system and infrastructure, and is subject to the same Tealium security and privacy protocols. For more information, see Data Storage and Access Controls. ... View more

This document describes General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) compliance as it relates to the Tealium Predict Machine Learning product. AudienceStream Compliance Predict is part of AudienceStream. With AudienceStream, your data is fully protected and is both GDPR and CCPA compliant, including advanced considerations such as opting in or out of automatic profiling. For detailed information about Tealium Consent Management compliance and Consent Management tools, see Consent Management. ... View more

This document describes how Predict data is collected and used. The Data Collection Stage Predict Models run at the end of each visit for every visitor at the normal time, which is during the end of the session aggregation stage. This is the stage at which data used by Predict is collected. At this time, the models produce a score – categorized as per Visit vs. per Visitor – that assigns a prediction percentage value between zero (0) and one (1). For additional information, see Data Storage and Access Controls. ... View more

This document describes where your data is stored during the data collection period and beyond, and the access controls used to protect that data. In this document: Table of Contents Placeholder Data Storage Predict data is stored using S3 Server-Side Encryption (SSE) and strict Identity and Access Management (IAM) policies to ensure that only necessary internal services can access the data. In the Machine Learning industry in general, the data produced by a model typically resides in a newly-created silo. The data remains siloed, even after the model is successfully integrated into an your systems and applications. For this reason, there is no guarantee that the data output by the model is fully integrated. Siloed data makes activation and the ability to achieve real business value difficult. The Tealium Predict product is implemented on each of the Tealium regional infrastructures to ensure that none of your data changes region when using the product. For example, if your general Tealium account uses an Australia-based infrastructure, then your usage of Predict occurs on that same infrastructure to ensure security. Data Usage and Sharing Data usage is consistent with your contracted terms for AudienceStream. Your data is only used for Tealium features. Data is not shared or sold, does not cross regions, and is protected by Tealium's general security and privacy protocols. GDPR and CCPA Compliance AudienceStream and your data is fully protected and compliant with the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Length of Time Data is Stored Using Predict, you can set the retention and storage period for related data when signing up for Predict. Contact your account manager if you have concerns about data retention periods. Data Access General usage of the Tealium Predict product does not require a third party or entity accessing or viewing your data. When your Predict packaging and services include assistance with data readiness, the Tealium internal data science team does have access to your models and training datasets to help ensure the quality of your machine learning models. Additional Information CCPA and GDPR Compliance Data Collected and Used by Predict ... View more

Visitor Stitching does not impact the Predict modeling results. Visitors vs. Visits Predict Machine Learning models train on visitor profiles, not visits. The actual and predicted values that display in the Model Training Details Confusion Matrix refer to the number of of visitors. Visitor Data Filtered When preparing for training, data is filtered by visitor_id . The model only trains on this (newest) stitched visitor_id and ignores the older (pre-stitched) ID and thus is not a problem for accounts and profiles with visitor stitching enabled. ... View more

This document describes machine learning technology concepts, goals, audiences, and technological advances. In this document: Table of Contents Placeholder Machine Learning Technology Machine learning is a subfield of artificial intelligence that focuses on enabling machines to learn without human guidance by recognizing patterns. Machine learning uses a set of predetermined rules to remember the patterns, analyze output, and create a model to explain the patterns and guide future behavior. In cases where you know what data you want, machine learning accelerates the path to acquiring the desired data. In cases where you do not know exactly what you want or need to identify a pattern, machine learning can find a pattern and reveal results that you can use to move forward with acquiring the data you need. For information about what machine learning is and what it is not, see Machine Learning vs. Artificial Intelligence. Goals Machine learning technology enables you to learn more about your users and, as a result, provide them more personalized, higher quality experiences while improving the efficiency of your marketing and advertising efforts. Audiences that Consume Machine Learning Technology Data scientists and developers who have a high level of knowledge about machine learning and actually work on machine learning projects are the primary consumers of machine learning tools and technologies. Everyone else, including marketers and business users, have a general understanding of the potential benefits of machine learning but lack the knowledge of how to start and implement a machine learning  project. Predict can be implemented and managed by members of this "everyone else" group. One does not have to be a Data Scientist or have a background in Data Science to get started and add immediate value to a business. Business Examples Machine learning models are commonly used by leading businesses to make predictions about the future behavior of their customers, such as estimating future purchase amounts. The most common business example used today is a recommendation engine, such as those built and used by Amazon, Netflix, AirBnb, and other large businesses. Advances Machine learning technology as a field has advanced to the point that it is being used successfully by some of the largest and most sophisticated technology companies. The public is generally aware of the fact that machine learning has large potential benefits for businesses and that it is here to stay. ... View more

This document provides a generic overview of the differences between Machine Learning and Artificial Intelligence. In this document: Table of Contents Placeholder Artificial Intelligence and Machine Learning Differences Artificial Intelligence (AI) and Machine Learning (ML) are correlated elements of computer science. Each of the two technologies are some of the highest trending in the industry today and used for creating intelligent software. Artificial Intelligence is the larger concept that is used create intelligent machines that can simulate human thinking capability and behavior. Machine Learning is an application or subset of AI that allows machines to learn from available data without being explicitly programmed to do so. About Artificial Intelligence Artificial intelligence is a field of computer science that enables a computer system to mimic or simulate human intelligence. The two words "Artificial" and "Intelligence" translate to "a human-made thinking power." An AI system does not require programming in advance. It uses algorithms that work with the AI intelligence already present in the system. This involves machine learning algorithms, such as "reinforcement learning" and "deep learning" neural networks. Some examples of AI implementations in used today include: Siri, Google AlphaGo, and various chess playing implementations. About Machine Learning Machine learning is a subfield of AI that enables machines to learn from historical data or experiences within a specific domain without being explicitly programmed. ML is reliant on extracting knowledge from the data. ML models are composed of complex algorithms. Through trial and error, the algorithms are refined to best explain patterns and best predict future outcomes and user behaviors. ML focuses on enabling machines or software to learn without human guidance, similar to how humans learn by recognizing patterns in the world and remembering the patterns through learned rules. Machines analyze information provided and create a model that explains the patterns. The information gathered can then guide future behavior. The primary benefit of ML is automation of effort. For example, in cases where you already know what you want to know, ML accelerates how quickly your goal is can be achieved. In cases where you do not know what you want to know or cannot identify a useful pattern in a dataset, ML can find a useful pattern and bring it the forefront for investigation. Some examples of ML implementations in use today include online recommendation systems, such as Google search algorithms, email spam filters, and Facebook automatic friend tagging suggestions. Additional Information Machine Learning Concepts and Technology ... View more

This document provides detailed information about model scoring techniques and formulas used to assign scores and rating to deployed models in the Tealium Predict product. In this document: Table of Contents Placeholder Target Attribute Health Ratings A target attribute is an AudienceStream attribute that defines or signals the visitor behavior that you seek to predict with any Predict model. The target attribute must be either a Boolean/Flag or a Badge type attribute and be Visit or Visitor-scoped. Healthy vs. Unhealthy Attributes When you create a model in Predict, potential target attributes from your AudienceStream dataset display. To the right of each attribute, a health rating of “Healthy” or “Unhealthy” displays. This health rating simplifies model creation by clarifying which of your flags and badges are ready or “healthy” and can be used to create models. A rating of “Unhealthy” does not necessarily mean that the attribute is problematic in other contexts. It simply means that this attribute is currently deemed insufficient for successful training of a Predict model. Machine learning technology in general requires a relatively high volume of data in order to succeed and Machine Learning models provide better results when trained on a large amount of data. Predict uses the following two factors to define Healthy vs. Unhealthy target attributes: The volume of data for the attribute The distribution of that volume between true and false values. Both the true and false groups must be above a minimum threshold. For example, for the dates that span the Training Date Range, the median daily counts of True and False visitors must be greater than or equal to 200. This threshold is intentionally set as low as possible to provide the most options possible for target attributes. A model with an Unhealthy target attribute will likely fail in training due to insufficient data for the model to learn on. If the target attribute you want to use is unhealthy, try one of the following common solutions: Wait for more data to accumulate. The problem often solves itself over a longer time period. Identify ways to drive additional traffic to the AudienceStream data source. Add additional data sources to your AudienceStream profile so that the volume of daily data increases for this target. Strength Scores (F1 Scores) An F1 Score is the typical metric used by experts to evaluate the quality of the type of Machine Learning models used by Predict. The F1 Score strikes a balance between two metrics: Precision and Recall. To calculate the F1 Score, Precision and Recall values are input into the following formula: F1 Score = 2 * ( (Precision * Recall) / (Precision + Recall) ) High Precision Model As an example, assume you have two colors of apples, red and green and your model seeks to predict which apples are red. If your model has high Precision, this means that the model is usually correct when it predicts an apple is red. If the model is creating a list of apples which are supposedly red, high Precision means that this list is mostly accurate and that the apples on the list are actually red. High Recall Model Using the same example, if your model has high Recall (Sensitivity), the model will be able to identify most of the red apples. A model with high recall does a good job of creating a thorough list of the red apples. Precision vs. Recall Analogy Using the same example of red and green apples, the following list describes expected results based on high or low Precision or Recall. High Precision, low Recall = a short list of red apples that is fairly accurate. High Precision, high recall = a longer list of red apples that is fairly accurate Low Precision, low Recall = a short list of red apples that is fairly inaccurate. This list contains more green apples. Low Precision, high Recall = a long list of red apples that is fairly inaccurate. This list contains green apples. The ideal model clearly should have both high precision and high recall. The concept of potential trade-offs (between the volume of apples predicted to be red, and the accuracy of those predictions) is a recurring theme that impacts how machine learning models are used in the real world. The Confusion Matrix A Confusion Matrix is a key tool used to evaluate a trained model. During the training and testing process that runs automatically when you create or retrain a model, Predict attempts to “classify” the visitors during the Training Date Range into two groups: true and false. These two groups reflect whether a user actually did the behavior signalled by the Target Attribute of your model, such as made a purchase or signed up for your email list. The Confusion Matrix allows you to easily view the accuracy of these predictions by comparing the true or false prediction value with the true or false actual value. There are four possible scenarios, as described quadrants description below. This comparison is made possible by the fact that the model is training on historical data (the Training Date Range). Once your model is deployed, the scenario changes. If your deployed model makes a prediction for a particular visitor today and the prediction timeframe is “in the next 10 days”, you will not know for up to 10 days whether the actual value ends up being true or false. Quadrants of the Confusion Matrix The four quadrants of the Confusion Matrix are: True Positive Correctly predicted true (predicted true and was actually true) True Negative Correctly predicted false (predicted false and was actually false) False Positive Incorrectly predicted true (predicted true but was actually false) False Negative Incorrectly predicted false (predicted false but was actually true) You can use the values of the quadrants to calculate the two constituent parts of F1 Score (Recall and Precision). Accuracy = (TP + TN) / (TP + FP + FN + TN) The accuracy is the number of correct predictions divided by the total number of predictions made. Precision Shows the proportion of all positive predictions that were positive. Recall Shows the proportion of positive predictions that were correct. The Confusion Matrix uses a threshold value of 0.5 to differentiate between predicted positive and predicted negative values. Formula Definition Reference The following list provides a descriptive reference of elements used in Predict modeling formulas referred to in the Confusion Matrix section and other portions of this document Visitor's Visit (Vn) The act of a Visitor visiting a website or the triggering of one or more data layer enriched events. Prediction Time Window (Wn) The Prediction Time Window starts at the end of each Visitor's visit, is measured in days, and is used to determine the Prediction Outcome. Prediction (Pn) The score generated by a Deployed Model. This score represents the likelihood that the Target Attribute Value will be set to True during the Visitor's next Visit (assuming the next visit occurs within the Prediction Time Window). This value is stored in the corresponding Prediction Attribute for the model. Prediction Attribute The numeric Visit scoped data-layer attribute that stores the Prediction value generated by a corresponding Deployed Model. The Prediction Attribute is created by default when a new model is created. Prediction Threshold (PT) The numeric threshold value selected to measure a Prediction value against the Target Attribute Value. For example, if the Prediction Threshold chosen is 0.5 and a Prediction value is set to 0.51, it is assumed that the Target Attribute Value will be set to True. Target Attribute The Flag or Badge attribute selected to represent the action to be predicted by the model. Target Attribute Value (AVn) The True or False value of the Target Attribute at the end of a visitor's visit. Prediction Outcome (O) The accuracy of the Prediction as determined by the Target Attribute Value, Prediction Threshold, and Prediction Time Window. If a Visitor returns within the Prediction Time Window, the Prediction Outcome is measured by the Target Attribute Value. If a visitor does not return within the chosen Prediction Time Window, the Target Attribute Value is ignored. Prediction Classification (PC) The value, True or False as a result of applying the Prediction Threshold to the Prediction. The ROC/AUC Curve In Tealium Predict, the ROC/AUC (under the curve) is a performance measurement reported for a trained model in the Model Explorer. In industry terms, the ROC is a true positive rate calculated as the number of true positives divided by the sum of the number of true positives and the number of false negatives. The ROC describes how well a model predicts the positive class when the actual outcome is positive. The true positive rate is also referred to as Sensitivity. The Receiver Operating Characteristics (ROC) curve and the area under this curve (referred to as AUC, for area under curve) are common tools in the machine learning community for evaluating the performance of a classification model. The ROC curve shows the trade-offs between different thresholds and consists of a plot of True Positive Rate (y-axis) against the False Positive Rate (x-axis). True Positive Rate = Sensitivity False Positive Rate = (1 - Specificity) Ideally, your will be able to distinguish between True and False classes. The model always predicts the correct answer. ROC Examples The Probability Distribution and ROC curve for a perfect model would look like the following: For the above ROC curve, AUC = 1. The entire graph area displays underneath and to the right of the red curve (boundary). For an extreme contrast, below is the Probability Distribution and ROC curve in scenarios where your model always predicts the wrong answer. Always labels True as False, and vice versa. For this ROC curve, AUC = 0. A poor scenario is defined as a model that is incapable of distinguishing between True and False classes. In this scenario, the Probability Distribution displays two large curves directly on top of each other. The ROC curve is a diagonal line; and AUC = 0.5. A realistic ROC curve, where 0.5 < AUC < 1.0 displays smaller values on the x-axis of the plot to indicate lower false positives and higher true negatives. Larger values display on the y-axis of the plot to indicate higher true positives and lower false negatives. When predicting a binary outcome, it is either a correct prediction (true positive) or not (false positive). Probability Distribution You can go to the Training Details panel for any version of any trained model to view a probability distribution of the predictions made by the model during training. The two colored curves of this chart represent the distributions of true and false predictions that the model made during training. Since the model training process uses historical data and you know whether each visitor actually performed the target behavior, it is possible to test the model by comparing the predictions for historical visitors versus the actual outcomes. The purpose of this comparison is to set aside a portion of the training dataset as the test subset. The probability distribution compares predictions against actual values for the visitors in the test subset. Visitors who were part of the True class (did perform the behavior) are displayed as part of the teal-colored curve and visitors who were part of the False class are part of the orange-colored curve. Ideal Probability Distribution The following list describes characteristics of an ideal probability distribution: There is clear separation between the True and False classes, as depicted by the teal and orange curves. This shows the model can easily and accurately classify visitors. Each class (curve) only covers one portion of the predicted value range. The False class us ideally focused in the lower range on the left and the True class is focused in the upper range on the right. Additional Information Data Readiness Deployed Model Health Scores ... View more

This document serves as a guideline of items to consider when creating audiences using your results from Tealium Predict. In this article: Table of Contents Placeholder Choosing Thresholds Once a model is deployed and making predictions, a common next step is to use your predictions to create one or more audiences. The prediction values in an AudienceStream output attribute for any model are decimal numbers in a defined range (0 – 1). The typical approach is to choose one or more thresholds or cutoff points to define which portion of that prediction range you want included in your audience. For example, if your model is predicting a likelihood to purchase in the next 3 days, you may want an audience of only the visitors who are most likely to purchase. To achieve this, you can create an audience using a rule that defines the threshold as 0.8 as an example. In this scenario, the audience will include only visitors with values greater than or equal to 0.8. There is no one objectively correct threshold value for all audiences, as different audiences serve different business goals and since there are usually substantial variations between models for different target behaviors or among different datasets. Sensitivity and Specificity To choose the best threshold, consider the potential trade-offs of choosing a threshold that is relatively high or low. This can be described as a trade-off between ending up with a larger audience that is less strict or a smaller audience that is laser-focused. In statistics terms, this is known as a trade-off between Sensitivity and Specificity. Sensitivity Sensitivity represents the True Positive Rate or how well the model correctly identifies all visitors who actually will purchase (using purchase as a the behavior your model is predicting). Specificity Specificity determines how well a model identifies false values. Specificity is the metric that evaluates a the ability of the model to predict true negatives of each available category. The True Negative Rate shows how well the model correctly identifies all visitors who actually will definitely not purchase (using purchase as a the behavior your model is predicting). ... View more

This document describes how access the Tiles view in Tealium Predict and use the sparkline graph as a guideline to view the ongoing health of a deployed model. In this article: Table of Contents Placeholder The Tiles View The Tiles view displays a "tile" for each model that displays a snapshot of the prediction results for that model. To access the Tiles view, go to Predict > Overview. From the Tiles view, you can view the status of a model, such as In Training or Deployed, in the upper right corner of each tile. Click any tile to view the details for a model and edit, if desired. The detailed summary in each tile displays the strength rating for the deployed model and a "sparkline graph" as an overview. Strength ratings are automatically recalculated automatically and the sparkline updates to include the latest rating. About Strength Score Ratings in the Tiles View In the Tiles view, the strength score rating displays in parentheses as a decimal number, such as "(0.91)". This number shows the F1 Score on which the rating is based. This strength rating is based on the F1 score, which cannot be known or calculated for a deployed model until the initial prediction timeframe has elapsed. For this reason, this strength rating is not available in the tile that displays for a particular model until the day after the prediction timeframe has elapsed. For example, if a model to predict the likelihood to purchase in the next 10 days is deployed on January 1st, the rating cannot be calculated until after January 10th. The Sparkline Graph The sparkline graph that displays in Tealium Predict is based on a simple trend graph. The graph displays an overview that depicts the ongoing health of a deployed model over time. Version Adaptation The sparkline graph adapts to versioning. For example, for a given model, if you first deploy Version 1 and later Undeploy Version 1 and Deploy Version 2, the graph automatically switches from Version 1 to Version 2. Adapting to versioning ensures you are provided with a continuous and realistic perspective of the strength of the output attribute for latest deployed version of the model. Additional Information See the following articles for additional information about model scores and evaluation: Evaluating the Trained Version Deployed Model Health Scores ... View more

This document provides an overview of how to evaluate your trained version before deploying your model. In this article: Table of Contents Placeholder Evaluate Model Status and Strength Use the following sections as a guide to view model status and the potential strength of predictions for your model before you deploy. View your Model Status Click on a model to view the details and status of the model or view the status for all models from the summary page. The goal prior to deploying is to have a model with a status of Trained. The following table describes possible model statuses: Status Description Requires Publish Model must be saved and published. In Training Model is currently in training but not yet trained. Waiting to be Trained Model is waiting to be trained. Save and publish to start training. Training Failed Model must be revised and retrained. Data Preparation Failed Data modification is required to make model viable. Trained Model is trained and ready to deploy. Currently Deployed Model is currently deployed and can be retrained or Undeployed. Evaluating the Strength Details of Trained Versions There are two types of strength ratings in Predict: static ratings for each trained version, and ongoing dynamic ratings for each deployed model. The strength rating assigned to each trained version of each model provides an easy-to-understand rating of the quality (strength) of the training and the model which resulted from it. Models not yet deployed are not yet assigned a Strength Score. Training is a one-time event and each retraining results in a new version number. For example, when you retrain Version 1, the resulting version is Version 2. Each retraining is a new and separate event, which makes this type of strength rating static and specific to each version. The rating for a version does not change over time. The quality of any model is a relative judgment, not an absolute fact. Different teams have different needs and goals for their models, as well as different levels of sophistication in their modeling and testing abilities and varying quality in their input datasets. For these reasons, model strength ratings are not regarded as absolute. The intention is to use the ratings as a general guideline for quality. Model Strength Scoring Model strength ratings for Predict provide an easy-to-understand rating of the quality (strength) of each version of each model. The rating system is comprised of four categorical labels. The labels are each based on an F1 Score, which is the typical metric used to evaluate the quality of a propensity model. The strength rating is visible next to each version on the Model Explorer page, in the Training Details panel, and in the Tiles view on the Overview page (for the latest trained version). The obvious goal is to achieve a higher strength score prior to deploying to ensure the most accurate predictions. F1 Score values are categorized using the following scale and trained models are assigned one of the following strengths: Poor – F1 score is less than or equal to 0.50 Fair – F1 score is greater than 0.50 but less than 0.60 Good – F1 score is greater than 0.60 but less than 0.80 Excellent – F1 score is greater than 0.80 The strength score contains a total score, the F1 Score, Recall, Precision, and Accuracy. For a detailed explanation of these scoring elements, see Evaluating Success. Next Steps Next steps are to retrain or deploy your model. To retrain your model in an attempt to increase the strength score prior to deploying, see Retraining a Model. To skip retraining and deploy, see Deploying a Model. ... View more

This document describes how to get started by selecting the AudienceStream output attributes you want to add or exclude for your data model. In this document: Table of Contents Placeholder Tealium Predict is an AudienceStream feature that evaluates visit and visitor-level attributes in AudienceStream. EventStream data is not evaluated. Build and Define Your Model To use Tealium Predict, the first step is to build and define a model. To do this, first determine which data and attributes of significance you want to gather or omit. Plain Language Indicators Predict displays information in plain language about each attribute in the drop-down list of available attributes that provides detailed information about the health of each attribute. This feature helps you determine, in advance, whether an attribute is a healthy or unhealthy candidate to use as an output attribute, as shown in the following example. Target Attributes A target attribute signals when an action has been performed. For example, a Boolean visit attribute named "Has Purchased" signals that a purchase event has occurred during a visit. These attributes will be Visit or Visitor-level Booleans. Exclusion Attributes Predict reviews each attribute in your AudienceStream profile automatically determines the top attributes that have a predictive relationship for the action you want to predict. Defining exclusion attributes is optional. To learn more about exclusion attributes, see Advanced Configuration Options. Get Started Go to Selecting Target Attributes to get started. ... View more

This document describes data wellness concepts and actionable steps you can take to examine and optimize the health of your data layer before starting with Predict. In this document: Table of Contents Placeholder Data Wellness Concepts This section defines basic data wellness concepts you can use to examine, audit, and optimize to improve the overall wellness of your data. The goal is to ensure that you start with a trusted data foundation. As data maturity levels differ between businesses based on a number of factors, it is important for organizations to review and consider each of the following concepts. After reviewing and understanding the concepts, advance to the data readiness steps. Defining your Strategy Define your business goals and your strategy. Define what do want to accomplish with machine learning technology, identify your audiences, and decide what data you want to capture and act on. Data Completeness Examine the volume and completeness of your data. As always, better data produces better outcomes. Governance and Consent Review and understand compliance practices and requirements regarding data usage. Accessibility Ensure that the insights you identify are accessible to other tools within your "tech stack". Data Integrity Review your AudienceStream implementation and compare the results to your the of data you want to collect, capture, and act on. To learn more about data readiness and review the wellness checklist, see the Data Readiness for Machine Learning Checklist. Data Wellness Services If you need help getting started, Tealium offers a Data Wellness services package to assist in getting your data ready to use with Predict. To learn how you can benefit from this service according to your unique data, contact your account representative. Data Readiness Actions The following list describes actions you and your teams can take to get your data foundation ready to add Predict Machine Learning. Use these steps as a general guideline to take the recommended actions to ensure optimal outcome from your Machine Learning models. Audit Audit your data layer and AudienceStream attributes. Examine your data quality, volume, and maturity and make any adjustments as needed to improve the quality and integrity of your customer data. Define and Integrate Define your digital data strategy and integrate. Your data foundation and infrastructure is implemented as vendor-neutral and can be used to power your entire "tech stack" without dependencies on any one vendor. Data Governance Practices Review, define, and implement your data governance practices. As compliance is ever-evolving, ensure that compliance issues are revisited on a regular basis to ensure the long-term health and viability of your data. Understand and Enrich Up until this point, your event data infrastructure has not yet acquired machine learning intelligence. Enrich and organize your incoming event data to improve the quality of the insights automatically produced AudienceStream. This step will enable you to immediately activate machine learning insights across your entire implementation, regardless of whether or not other components of your implementation have machine learning capabilities. Orchestrate After your infrastructure is defined and your data insights are built into the way your business collect data, you can orchestrate these insights to drive action in any integrated tool of implementation. You can additionally implement your own business rules that will automate data insights to flow across all your customer engagement channels in real-time Best Practices For optimal results, Tealium recommends training using a prediction timeframe that contains 90 or more days of data. In many circumstances and depending on your target use case, you can successfully train models using a smaller date range and less data. As a minimum, you must train a model on at least as many days as your prediction timeframe. Get Started After reviewing the Getting Started and Defining your Strategy and Goals sections, you can now move on to Creating a Model to get started. ... View more

This document provides a descriptions of various approaches you can use to define your strategy and goals before you begin creating and training models. In this document: Table of Contents Placeholder The variety of approaches you can use to define your strategy and goals using Machine Learning is virtually endless. For example, you can augment existing use cases, create exploratory models, or create specific use cases for targeted findings. Implementation If you have AudienceStream, you have the ability to begin using Predict Machine Learning. There are no further requirements. For additional information, see Prerequisites. As a general approach to get started, train one or more models and review the Included Attributes on the Training Details report. The details in this report can provide you with a source of new insights about which AudienceStream data points are the key drivers to an important user event. You can also use Predict as a tool for learning more about your data, your implementation, and as a source of information for new ideas. Best Practices The following general best practices can provide more robust results: Define the problem you want to solve and goals to achieve resolution. Establish best practices regarding collecting and cleaning your data before you begin. For more information, see Data Readiness. During your data readiness stage, join any siloed datasets and consider other characteristics of your organization's data that can be refined. Where possible, use longer date ranges for training. Consider ways to use Predict to improve or augment existing audiences and make them more efficient. Ensure that your AudienceStream profile has high-quality attributes to signal target behaviors, such as Badges or Booleans. If they are not present, create the attributes as soon as possible to allow more time for data to accumulate. Deploy your model in a production environment or real-world application. Evaluate how well your model is working in production and return to the training and testing stages as needed for improvements. Use Cases To help jump start your implementation, you can review popular and emerging machine learning use cases being implemented by Tealium customers. From here you can gather ideas or implement proven use cases, such as: Proactive Marketing: Machine Learning-Powered Segmentation Marketing Efficiency: Improve Purchase or Conversion Rate Customer Retention: Reduce Churn Customer Experience: Funnel Optimization Predictive Analytics: Get Customer Insights and Validate Assumptions For detailed information, see Tealium Predict ML: Top 5 Emerging Machine Learning Use Cases with Customer Data. ... View more

This section describes what is required to use the Tealium Predict product, suggested steps to take before you begin to ensure ideal results, and available services. Required AudienceStream To get started using Predict, all you need to have is the Tealium AudienceStream product with data collection enabled. Data Collection To ensure that you have enough data to begin building meaningful models using past data, data collection has already been enabled for AudienceStream for a minimum of 30 days prior to the feature release. For additional information, see Data Collected and Used by Predict and Data Storage and Access Controls. Suggested For optimal results, you need a high-quality dataset, which consists of a clean data layer with a large amount of training data. Better data layer health allows you to create models that can immediately provide meaningful prediction results. For additional information, see Data Readiness for Machine Learning. Optional Services Tealium Predict is a product, not a service. Though no coding or data science knowledge is required to use the product, you can improve both your experience and your results by improving your overall AudienceStream implementation. If you would like help assessing the current state of your data layer, Tealium can provide professional services centered around your current AudienceStream implementation and overall data wellness. For additional information about data readiness and data wellness services, see Data Readiness. ... View more

This document describes the Tealium Predict product tiers and availability based on the number of models created, used, and deployed. In this document: Table of Contents Placeholder If you have the Tealium AudienceStream product, you automatically have access to a "test drive" of Tealium Predict. Review this document to get started with your first model. To continue after the trial tier is used, contact your account representative to begin planning for additional models to expand your findings and continue growth. Trial Tier The trail tier for Predict Machine Learning includes includes one (1) deployed model and five (5) trained versions of the model to use as you see fit. With the Tealium AudienceStream product, you can immediately access your trial period. Having a deployed model will allow you to view and analyze end-to-end usage of Predict and plan for next steps. To start, simply add your first model. When you upgrade, you can continue to use the model you created and deployed in the trial tier. Suggested Starting Points Though you can use the trail period for any type of model you desire, the following list provides suggested starting points for your first model: Predict which users are good targets (high likelihood), or bad targets (low likelihood), based on your ultimate goal by identifying users with a high likelihood for conversion and optimizing your budget for those users. Identify users with a low likelihood for conversion and exclude those users. Explore insights for a customer attribute that is already defined, such as making a prediction and see which data points correlate with your goal. Beyond the Trial Beyond the trail period, Predict is a paid feature of AudienceStream, which must be contracted. Once you use your "free" trained versions, training is disabled. Your models will still work, but model accuracy decline for that model without continued training to refresh your data. model. To create more models and continue with training, contact your account representative. Usage and Billing You do not automatically incur charges if you attempt to used Predict beyond the trial period. To incur billing, you must have a signed contract. The cost of Predict is based on volume tiers and the number of model trainings. ... View more

This document defines general statistical modeling terminology, terms specific to Tealium products, and terms used in the Tealium Predict interface. In this article: Table of Contents Placeholder Attributes In Tealium products, attributes allow you to define important characteristics that represent visitor habits, preferences, actions, and engagements with your brand. Defining attributes in your data layer as the first step in implementing tag management. In the Tealium Predict product, you must determine which of these attributes (target attributes) you want to add or exclude from your data model. Audience In Tealium products, an audience is defined as a group of visitor profiles that share a set of attribute conditions and used to trigger vendor actions (connectors) in real-time. The more conditions used to create an audience, the more specific your audience. AudienceStream AudienceStream is a Tealium product consisting of Omnichannel customer segmentation and a real-time action engine. AudienceStream takes the data that flows into the Tealium EventStream product and creates visitor profiles that represent the most important attributes of a visitor engagements with your brand. Visitor profiles are then segmented by shared behaviors to create targeted audiences to fuel your marketing technology stack in real-time via connectors. Confusion Matrix In Tealium Predict, the confusion matrix, also known as an error matrix, is a performance measurement reported for a trained model in the Model Explorer that compares actual and predicted values. In industry terms, a confusion matrix uses a set of test data for which true values are known and then displays actual and predicted values in table format to allow you to visualize the performance of a given algorithm. Data Collection In Tealium Predict, data collection refers to a predetermined time range in which data is collected to use with the product. Data Layer In the Tealium iQ Tag Management product, a data layer is the foundation of tag management that defines the attributes of your website, such as site language or page names, and other user behaviors that you want to track, such as purchases and logins. Data Scientist In industry terms, a Data Scientist is typically an analytical expert that utilizes skills in technology and social science to look for trends and manage data using industry knowledge, contextual understanding, and skepticism of existing assumptions to reveal solutions to business challenges. Data Source In Tealium products, a data source is a system that sends data to EventStream, such as a website, a native mobile application, or any custom application that can report event data. Data sources are platform-specific and provide the necessary code and instructions to complete an installation. Examples: Android, Apple TV, HTTP API, Java, JavaScript for Web (iQ TMS), Swift (iOS), Python, Roku. Deployed Model In Tealium Predict, a deployed model is a model that has been "trained" and then deployed to populate prediction values into your customer profiles in AudienceStream. Hyperparameter (Tuning) A hyperparameter is a parameter that is set before the learning process for a model begins. These parameters are tunable and can directly affect how well a model trains. Machine Learning Machine learning is a subfield of artificial intelligence that focuses on enabling machines to learn without human guidance by recognizing patterns. Machine learning uses a set of predetermined rules to remember the patterns, analyze output, and create a model to explain the patterns and guide the future behavior. In cases where you know what data you want, machine learning accelerates the path to acquiring the desired data. In cases where you to not know exactly what you want or a pattern to identify, machine learning can find a pattern and reveal results that you can use to move forward with acquiring the data you need. Model In Tealium Predict, a model represents the behavior you are predicting within a specific timeframe, such as a purchase, conversion, or any customer behavior tracked in AudienceStream. A model is created using an algorithm and the results are used to explain patterns and predict future outcomes. Model Explorer In Tealium Predict, the Model Explorer is an interactive section of the product interface where you can view performance measurements for each model in each stage and fine-tune the model with actionable items from the interface, such as retrain or deploy. Model Strength In Tealium Predict, the model strength is a combination of metrics and acceptable thresholds used to assign a score to a model. These scores are used to determine model quality and performance and the ability of the model to perform in the real world. The model score displays in the Model Explorer as "Excellent", "Good", "Fair", or "Poor". Probability Distribution (Returning Visitor) In Tealium Predict, the probability distribution is a performance measurement reported for a trained model in the Model Explorer that refers specifically to returning visitors. In industry terms, this is a mathematical function whose outcome is to provide the probabilities of the occurrence of different outcomes of an experiment, and thus the probability of a predetermined event to occur. Receiver Operating Characteristics (ROC) In Tealium Predict, the ROC/ AUC (under the curve) is a performance measurement reported for a trained model in the Model Explorer. In industry terms, the ROC is a true positive rate calculated as the number of true positives divided by the sum of the number of true positives and the number of false negatives. An ROC describes how well a model predicts the positive class when the actual outcome is positive. The true positive rate is also referred to as sensitivity. Retrain Over time, and without additional training, the prediction accuracy for a model is degraded. When you retrain a Tealium Predict model with new data, prediction accuracy is increased and remains more accurate over a longer period of time Training In Tealium Predict, training refers to the stage in which a model is consuming and analyzing data for a predetermined period of time to be used for predictions. The size and quality of the data used during this stage plays an important part in the accuracy of results when you deploy the model. Trained Version In Tealium Predict, a trained version is a single instance of training a model. Every machine learning model is trained with data in order to accurately make predictions. ... View more

This article describes how to set up the Hotjar tag in your Tealium iQ Tag Management account. In this article: Table of Contents Placeholder Tag Tips Use mapping to dynamically override the standard config values. Tag Configuration First, go to Tealium's tag marketplace and add the Hotjar tag (Learn more about how to add a tag). After adding the tag, configure the following settings: Hotjar ID Your Hotjar Site ID. Hotjar Snippet Version The version of your tracking code from Hotjar. Add Tag from Code (Optional) Use the following steps to add the Hotjar tag from code" Log in to your Hotjar account. From the account dashboard, click Install code manually. Click Copy to clipboard. Log in to Tealium iQ Tag Management, go to iQ Tag Management > Tags. Click Add Tag. Click Detect Tag from Code. Paste the code you copied in Step 3. Data Mappings Mapping is the process of sending data from a data layer variable to the corresponding destination variable of the vendor tag. For instructions on how to map a variable to a tag destination, see Data Mappings. The available categories are: Standard Variable Description hjid Hotjar ID hjsv Hotjar Snippet Version ... View more