Importing Offline Data (Omnichannel)

Importing Offline Data (Omnichannel)

by on ‎08-27-2015 08:19 PM - edited 3 weeks ago by Community Manager (6,648 Views)

Omnichannel is a feature for importing offline data into the Universal Data Hub (UDH) where it enriches visitor profile data and becomes actionable within connectors.

This guide has three sections:

  • Before You Begin - Learn about preparing your data for a successul import into the UDH.
  • How It Works - Learn about the process: from the basic components and import process to the final results within the UDH.
  • Using Omnichannel - Step by step instructions on configuring your omnichannel imports.

Here's a quick overview of the Omnichannel feature: 

This guide covers the following topics:

Table of Contents Placeholder

Before You Begin

The use of omnichannel imports requires that your account be enabled for AudienceStream. The following topics will help you get started with omnichannel imports to ensure a successful import.

What Data to Import

The best data to import will help create a fuller and richer visitor profile of your customers. To get the most out of an omnichannel import the data should have a visitor identifier associated with each row of data. This will ensure that the corresponding visitor within the Universal Data Hub is enriched.

Here are some recommended data sets for use with omnichannel imports:

  • Demographic Data
  • Prospect/Lead Data
  • Historical Purchase/Return Data
  • Offline Interactions (Support, Direct Marketing, Event Attendance, etc.)

File Format

To ensure a successful import of the data it's critical to understand the expected file format. Omnichannel supports CSV (comma separated values) files where the first line of the file must be a header line that names the columns of the file. Each line after that represents an event or a visitor record and must contain at least one visitor identifier attribute.

Column names may not contain "#", "^", or whitespace characters.

Files can be compressed into zip files to minimize upload time. The system will detect zip files automatically and handle them accordingly.

File Naming

The CSV files must be named using the following format: {prefix}_{version}.csv. This format consists of two parts:

  • prefix - a unique identifier for groups of files that share the same CSV column names.
  • version - a unique identifier for a file within a prefix, usually a timestamp and (optionally) a version number.

Example Filenames

In-store purchases from March 19, 2017, separated into two files ("a" and "b").


In-store returns from March 14 and March 15, 2017.


File Size and Import Limits

You can import up to 1,000,000 rows of data per day. The maximum number of rows per file is 100,000. Therefore, you can import the maximum amount of data by creating 10 files each with 100,000 rows for a total of 1,000,000 rows.

Smaller files, no bigger than 50MB, provide optimal import performance.

Splitting Large Files Into Smaller Files

If a CSV file is larger than 100,000 rows it must be split into smaller files, each containing the header line from the original. In this example, a file named master_purchases.csv has 325,000 rows of purchase data and is split into smaller files, each with 100,000 rows.

The split command creates smaller files with a lexically ordered suffix using the characters a-z.

$ split -l 100000 -a 1 master_purchases.csv purchases-
$ ls
master_purchases.csv  purchases-a  purchases-b  purchases-c  purchases-d

The new files would have the following number of rows:

  • purchases-a : 100,000
  • purchases-b : 100,000
  • purchases-c : 100,000
  • purchases-d :  25,000

Then the head and cat commands are used to add the header line from the original file to the smaller files.

$ for file in purchases-*
> do
> head -n 1 master_purchases.csv > tmp
> cat $file >> tmp
> mv -f tmp $file
> done

Data Format

The data within your files should follow these guidelines to ensure a successful import:

  • Group Rows by Visitor ID
    Each row will have a visitor ID field and the file may contain multiple lines for the same visitor. Lines that contain the same visitor ID should be grouped together to optimize the import process.
  • File encoding

We suggest utf-8.

  • Field Ordering

The ordering of the fields does not matter, apart from the visitor ID column, which we strongly suggest is leftmost.

  • Extra Columns

Having extra fields that are not configured in AS will mean AS ignores those extra fields, but processes the configured fields.  Note that fields names must match the configuration exactly, inlcuding case.

  • Missing Columns

Having missing fields in the file (fields that AS has been configured to process) will mean AS ignores the missing fields and processes the configured fields that are present. The only field which is mandatory is the visitor ID field.  You may use rules in AS when enriching other attributes to check for the presence or absence of omnichannel fields to effectively make other fields mandatory, if you wish.

  • Normalize "Empty" Values

Some rows might not contain a value for each column. Omnichannel expects these columns to be empty and not to contain placeholder values such as "null", "empty", "blank", etc.

  • Normalize Date/Time Values
    Date/time values must be consistent throughout a file for a given field. Each date/time value must match the format specified in your omnichannel configuration for that field. For example, if the expected format is "yyyy-MM-dd", the values must contain 4-digit years and 2-digit month and day values eg. "2016-02-09".
    The following date components are supported: Year, Month, Day, Hour, Minutes, Seconds, Milliseconds, AM/PM, ISO , RFC, Era.
  • Double-Quote Values with Commas
    If a value contains a comma (,) it must be surrounded by double-quotes to ensure the integrity of the CSV columns. Putting all values in double-quotes is highly recommended. Use a doubled double quote to indicate a double quote character in the data.
  • Omit Special Characters from Currency Amounts

Columns that represent currency amounts, such as "OrderTotal" or "ProductPrice", cannot contain currency symbols or commas.

Valid Values Invalid Values           
  • 39.75
  • 0
  • 1399.00
  • $39.75
  • zero
  • €1.399,00

How It Works

The primary goal of an omnichannel import is to supplement online visitor profile data with valuable offline data. This is accomplished using two types of attributes: visitor and omnichannel. Visitor attributes define your visitor profiles and represent the unified view of your customer. Imported omnichannel attributes help complete this view of the customer via attribute enrichments. An omnichannel attribute must be used in an enrichment of a visitor attribute in order for its data to be imported.

There are two components used in setting up an omnichannel file import: a file transfer service and a file definition.

File Transfer Service

The file transfer service is a secure location where you upload your files for Tealium to retreive them. Tealium supports the following file transfer services:

  • Amazon S3 (Tealium bucket or your own bucket)
  • Microsoft Azure File/Blob Storage

If using your own file transfer service, be sure to have the connection details ready before proceeding.

File Definition

A file definition is a detailed description of the CSV files to import. This includes the names of the files and the name of each CSV column (corresponds to a UDH attribute). Each file definition must also specify a visitor identifier that will appear in each row of the file. This identifier will link the imported data to visitor profiles in AudienceStream.

Visitor Enrichment

When a file is processed, the omnichannel attributes are used to create and update visitor attributes via enrichments. For this reason, it's important that your visitor attributes are configured to use the new omnichannel attributes in their enrichments.

For example, if you are importing offline purchase data and want to update a visitor lifetime order value, you will create an enrichment in the visitor attribute to leverage the omnichannel attribute.

  • Omnichannel Attribute: order_total_amount
  • Visitor Attribute: Lifetime Order Value

Resulting enrichment:


Import Process

Once a file transfer service is configured with a file definition you will upload a file to the service. The omnichannel system then follows this order of operations:

  1. Check for New Files
    The system checks the file transfer service for new files every 10 minutes.
  2. Copy New Files
    When a new file is detected it is copied from the file transfer location and processed in the UDH.
  3. Match Filename Prefix to File Definition
    The prefix of the filename is used to identify which file definition to use when importing the data in the file. The file definition is used to identify which visitor ID attribute to use and which data attributes to import.
  4. Process Files
    The header line is read to identify the attributes being ingested. From there the following processing is performed:
    • Visitor Lookup: the visitor ID is used for a lookup of the visitor record in AudienceStream. If an existing visitor record is not found a new one is created.

      Grouping rows with the same visitor ID will increase the speed of the import.

    • Attribute Enrichment: the visitor record will be enriched according to the attributes imported and the existing enrichments in your account.
    • EventStream/DataAccess Processing: If configured to do so, the imported data will be made available to EventStream connectors and stored in DataAccess.

Using Omnichannel

Setting up an omnichannel enrichment requires you to configure a file transfer service and a file definition. To begin setting up an omnichannel enrichment, go to Enrich > Omnichannel and click + Add Enrichment. The omnichannel dialog will appear with a two-step process to configure a file transfer service and create a file definition.

Adding a File Transfer Service

To configure your file transfer service:

  1. Select a Service. The remaining fields will change depending on the service chosen.
  2. Enter a Title to identify this enrichment, preferably by its channel e.g. "In-Store Transactions"
  3. Enter the credentials necessary to access the service you selected. If you selected Tealium S3 Bucket, then the necessary credentials are generated for you automatically.
  4. (Optional) Enter notes to describe the enrichment.
  5. Click Next to proceed to file definitions.

Select Service uploading file to s3.png

Adding a File Definition

A file definition identifies the names of the columns in the CSV file. These column names automatically generate omnichannel attributes with the same names. In the next section these attributes are used to enrich visitor attributes.

To set up a file definition and define omnichannel attributes:

  1. Click + Add File Definition. The file definition form appears.
  2. Send to DataAccess and EventStream Connectors - Check this box if the imported data should be made available to EventStream connectors and DataAccess storage.
  3. Unique File Prefix - Enter a prefix to identify these files. This prefix identifies which files on the file transfer service to retreive.
  4. Visitor ID - Select a Visitor ID attribute from the Map drop-down. In the From text field, enter the name of the corresponding column from the CSV file.
  5. Omnichannel Attributes (CSV Column Names) - Enter the name of each column to import from the file. Each column creates a new Omnichannel Attribute with the same name, which is made available for use in rules, enrichments, and streams.

    Column names are case sensitive and may not contain "#", "^", or whitespace characters.

    • Date Format - Columns with date/time values must have a matching date format setting. Click the calendar icon and select a data format e.g. "yyyy-MM-dd"
    • Restricted Data - Check this box to mark a column as having restricted data and to exclude it from other Tealium services (for more information, see What is Restricted Data?).
  6. Click Save.


Download/Copy Sample CSV 

To see an example of what your CSV file should look like based on the file definition, click Download CSV or Copy CSV. The resulting file will contain two rows of data, the header line with the column names and one row of placeholder data.

Example sample CSV:


Using Omnichannel Attributes

The next step is to configure your visitor attributes with enrichments based on the omnichannel attributes. Omnichannel attributes appear in the attribute drop-down menus for rules, enrichments, and streams.

To view the list of existing omnichannel attributes:

  1. Navigate to Enrich > Attributes.
  2. In the side bar, select the Omnichannel scope.omnichannel-attributes-list.png

To enrich a visitor attribute with an omnichannel attribute:

  1. Navigate to Enrich > Attributes.
  2. In the side panel, under Scope, select Visitor.
  3. Select a visitor attribute to enrich.
  4. Add an enrichment. The omnichannel attributes are available from the attribute drop-down menus for use within the enrichment.

Uploading Ominchannel Files

The final step is to upload your CSV files. You will require a third-party application to initiate the upload. We recommend Cyberduck because it’s free and supports FTP and Amazon S3, but you may use any client for this purpose.

To upload a file via FTP or Amazon S3 using Cyberduck:

  1. Install (or launch) Cyberduck.
  2. Create a new connection and give it a title.
  3. From the first dropdown, select the file transfer service used in your Omnichannel configuration.
  4. Enter the credentials (username and password) for your service.
    • My FTP Connection and My SFTP Connection
    • Tealium S3 Bucket - the credentials are pre-populated as Access Key (username) and Secret Key (password).
  5. Provide other details, such as the server, path, port, etc, required by your service.
    Read more about accessing third-party S3 buckets using Cyberduck.
  6. Save the connection.

Now you are ready to upload files by dragging and dropping your CSV files into Cyberduck. 

Additional Resources

Frequently Asked Questions 

What if the file transfer service fails?

If AudienceStream fails to copy a file via the file transfer service it will simply wait 10 minutes and try again.

What if a file failed to be processed?

When a process failure occurs, the file will be ignored and not attempted to be processed again. Please submit a Support Desk request to get assistance diagnosing a problem with a failed import.

File processing can fail for one of the following reasons:

  • the CSV file is improperly formatted eg. not a valid CSV file
  • the column names used in the File Definition do not exist in the file