I'm assuming you are asking about IE8? Based on http://stackoverflow.com/questions/1328963/detect-ie8-compatibility-mode
there are two ways.
1) IE8 includes a trident token in the User-Agent string regardless of compatibility mode.
See MSDN for more details: http://blogs.msdn.com/ie/archive/2009/01/09/the-internet-explorer-8-user-agent-string-updated-editio...
IE7 on Windows Vista
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
IE8 on Windows Vista (Compatibility View)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0)
IE8 on Windows Vista
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)
Please let me know if this helps.