I'm assuming you are asking about IE8? Based on http://stackoverflow.com/questions/1328963/detect-ie8-compatibility-mode there are two ways.
1) IE8 includes a trident token in the User-Agent string regardless of compatibility mode.
See MSDN for more details: http://blogs.msdn.com/ie/archive/2009/01/09/the-internet-explorer-8-user-agent-string-updated-editio...
IE7 on Windows Vista
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
IE8 on Windows Vista (Compatibility View)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0)
IE8 on Windows Vista
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)
Please let me know if this helps.
Gotcha. I just did a search for "js detect ie compatibility mode" and found this post to github https://gist.github.com/jasongaylord/5733469. I think this is up the alley of what you are looking for. It's checking to ensure jQuery is installed, captures the browser and if compatibility mode is on, and displays some text.