Filtering Bot traffic from Tealium Collect data

[bmarcus]

I'm new to Tealium Collect to capture data into Audience Stream. Is there a way to exclude known BOT traffic and our internal traffic from collection?

[GavinAttard]

Hi

Have you tried doing some analysis on user agent, then using some patterns in a condition to prevent the collect tag from firing?

kr

[kara_nichols · Accepted solution]

Tealium automatically filters events for bots based on the user-agent when data is sent via the Collect tag. If such filtering is performed, the i.gif network request for the Collect tag will have a X-Error: bot request header in the response.

Below are user agents that we recognize as bots, if they contain any of the following:

robot
spider200PleaseBot
360Spider
4seohuntBot
80legs
AdsBot
AhrefsBot
AlertSite
Alexibot
Applebot
atSpider
autoemailspider
Baiduspider
BecomeBot
Bingbot
BingPreview
Black\.Hole
Black Hole
BLEXBot
BlowFish
Bullseye
CatchBot
Catchpoint
CCBot
Cheesebot
citeseerxbot
ContactBot
ContentSmartz
crawler4j
DataCha0s
datagnionbot
DBrowse
DotBot
DuckDuckBot
EmailSiphon
EmailSpider
envolk
Exabot
Facebot
facebookexternalhit
FAST-WebCrawler
FAST Enterprise Crawler
Feedfetcher-Google
Genieo
Gid_Synthetic
gigablast
Gigabot
GingerCrawler
Girafabot
GomezAgent
Googlebot
ia_archiver
ips-agent
Keynote
KHTE
KXTN
linkdexbot
LinkedInBot
Mediapartners-Google
MetaJobBot
Microsearch
MJ12bot
Mnogosearch
msnbot
MSRBot
NaverBot
oBot
PagePeeker
Pingdom
proximic
purebot
Psbot
redditbot
RU_bot
Scanbot
ScoutJet
Scrapy
SeznamBot
SimpleCrawler
SimplePie
Site24x7
SiteLockSpider
Slackbot
Slurp
Sosospider
Sougou
spbot
Speedy Spider
SPEng
Twiceler
Twitterbot
UnwindFetchor
Veoozbot
Voilabot
voyager
WebDataCentreBot
Webmetrics
WhatsApp
Yandex
yanga
Yodaobot
YottaaMonitor
Youdao
zibber
ZyBorg

To filter additional internal and/or bot traffic, you could add load rules on the Collect tag in Tealium iQ to prevent the event from being sent server-side.

There is also a server-side option to filter traffic specifically from AudienceStream, which could be done based on User Agent and/or IP address rules. For the server-side approach, please contact a Tealium technical resource to help enable the proper settings.

[bmarcus]

Thanks so much, Kara!!

[mountaintoph]

You should add in traffic uptime solutions like StatusCake:

userAgent: StatusCake_Pagespeed_Indev