User-Agent detection, history and checklist

History

User-Agent: <something> is a string of characters sent by HTTP clients (browsers, bots, calendar applications, etc.) for each individual HTTP request to a server. The HTTP Protocol as defined in 1991 didn’t have this field, but the next version defined in 1992 added User-Agent in the HTTP requests headers. Its syntax was defined as “the software product name, with an optional slash and version designator“. The prose already invited people to use it for analytics and identify the products with implementation issues.

This line if present gives the software program used by the original client. This is for statistical purposes and the tracing of protocol violations. It should be included.

Fast forward to August 2013, the HTTP/1.1 specification is being revised and also defines User-Agent.

A user agent SHOULD NOT generate a User-Agent field containing
needlessly fine-grained detail and SHOULD limit the addition of
subproducts by third parties. Overly long and detailed User-Agent
field values increase request latency and the risk of a user being
identified against their wishes (“fingerprinting”).

Likewise, implementations are encouraged not to use the product
tokens of other implementations in order to declare compatibility
with them, as this circumvents the purpose of the field. If a user
agent masquerades as a different user agent, recipients can assume
that the user intentionally desires to see responses tailored for
that identified user agent, even if they might not work as well for
the actual user agent being used.

Basically, the HTTP specification discouraged since its inception the detection of the User-Agent string for tailoring the user experience. Currently, the user agent strings have become overly long. They are abused in every possible way. They include detailed information. They lie about what they really are and they are used for branding and advertising the devices they run on.

User-Agent Detection

User agent detection (or sniffing) is the mechanism used for parsing the User-Agent string and inferring physical and applicative properties about the device and its browser. But let get the record straight. User-Agent sniffing is a future fail strategy. By design, you will detect only what is known, not what will come. The space of small devices (smartphones, feature phones, tablets, watches, arduino, etc.) is a very fast-paced evolving space. The diversity in terms of physical characteristics will only increase. Updating databases and algorithms for identifying correctly is a very high maintenance task which is doomed to fail at a point in the future. Sites get abandoned, libraries are not maintained and Web sites will break just because they were not planned for the future coming devices. All of these have costs in resources and branding.

New solutions are being developed for helping people to adjust the user experience depending on the capabilities of the products, not its name. Responsive design helps to create Web sites that are adjusting for different screen sizes. Each time you detect a product or a feature, it is important to thoroughly understand why you are trying to detect this feature. You could fall in the same traps as the ones existing with user agent detection algorithms.

We have to deal on a daily basis with abusive user agent detection blocking Firefox OS and/or Firefox on Android. It is not only Mozilla products, every product and brand has to deal at a point with the fact to be excluded because they didn’t have the right token to pass an ill-coded algorithm. User agent detection leads to situation where a new player can hardly enter the market even if it has the right set of technologies. Remember that there are huge benefits to create a system which is resilient to many situations.

Some companies will be using the User-Agent string as an identifier for bypassing a pay-wall or offering specific content for a group of users during a marketing campaign. It seems to be an easy solution at first but it creates an environment easy to by-pass in spoofing the user agent.

Firefox and Mobile

Firefox OS and Firefox on Android have very simple documented User-Agent strings.

Firefox OS

Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

Firefox on Android

Mozilla/5.0 (Android; Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

The most current case of user agent detection is to know if the device is a mobile to redirect the browser to a dedicated Web site tailored with mobile content. We recommend you to limit your detection to the simplest possible string by matching the substring mobi in lowercase.

/mobi/i

If you are detecting on the client side with JavaScript, one possibility among many would be to do:

// Put the User Agent string in lowercase
var ua = navigator.userAgent.toLowerCase();
// Better to test on mobi than mobile (Firefox, Opera, IE)

if (/mobi/i.test(ua)) {

    // do something here

} else {

    // if not identified, still do something useful

}

You might want to add more than one token in the if statement.

/mobi|android|touch|mini/i.test(ua)

Remember that whatever the number of tokens you put there, you will fail at a point in the future. Some devices will not have JavaScript, will not have the right token. The pattern or the length of the token was not as you had initially planned. The stones on the path are plenty, choose the way of the simplicity.

Summary: UA detection Checklist Zen

Do not detect user agent strings
Use responsive design for your new mobile sites (media queries)
If you are using a specific feature, use feature detections to enhance, not block
And if finally you are using UA detection, just go with the most simple and generic strings.
Always provide a working fallback whatever the solutions you chose are.

Practice. Learn. Imagine. Modify. And start again. There will be many road blocks on the way depending on the context, the business requirements, the social infrastructure of your own company. Keep this checklist close to you and give the Web to more people.

Technical Evangelist & Editor of Mozilla Hacks. Gives talks & blogs about HTML5, JavaScript & the Open Web. Robert is a strong believer in HTML5 and the Open Web and has been working since 1999 with Front End development for the web - in Sweden and in New York City. He regularly also blogs at http://robertnyman.com and loves to travel and meet people.

More articles by Robert Nyman [Editor emeritus]…

24 comments

Ron

Unfortunately, there are so many old user agents that break these rules that I fear User Agent Strings will always be a useless aspect of practical HTTP.

September 12th, 2013 at 03:25
1. Ronan Cremin
  
  Do remember that your UA string is being used countless times per day to improve your experience by the likes of Google, Facebook, eBay, Amazon, Netflix etc. All of these major brands fine-tune the user experience based on it, but do it well enough that most people don’t even notice. Taking Google as an example, their homepage /looks/ the same on all devices but in fact varies massively depending on what you’re using.
  
  In practice the older UA strings don’t cause a problem to good implementations—they handle the edge cases just fine.
  
  September 16th, 2013 at 03:48
  1. Karl Dubost
    
    @Ronan “All of these major brands fine-tune the user experience based on it, but do it well enough that most people don’t even notice.”
    
    That’s true. The issue is with the other part which is not in the most people ;) see for example http://mzl.la/164Mj9x That’s the issue which is at stake. It’s not about the working cases but about the many times it fails. Each time it fails because a person had a different device, with different capabilities or the “wrong” brand.
    
    September 16th, 2013 at 08:10
    1. Ronan Cremin
      
      Bad implementation != bad idea
      
      I continue to enjoy wine despite having a bad bottle from time to time.
      
      I would argue that people are overwhelmingly benefitting from this rather than getting hurt by it. There’s a “silent evidence” thing going on here—one notices the failures but not the successes.
      
      (BTW, I don’t mean to hijack this entire thread but I think it’s good to hear dissenting opinions)
      
      September 16th, 2013 at 08:35
Olivier Mengué

Related:
– « Introducing IE9’s User-Agent string » https://blogs.msdn.com/b/ie/archive/2010/03/23/introducing-ie9-s-user-agent-string.aspx?Redirected=true
– « History of the user-agent string » http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/

September 12th, 2013 at 04:39
Ronan

> Basically, the HTTP specification discouraged since its inception the detection of the User-Agent string for tailoring the user experience.

Really? Take look at the specs. RFC 1945 (HTTP 1.0) says the following:

“The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations.”

RFC 2616 (HTTP 1.1) says the same thing.

Two other points:

> sites will break just because they were not planned for the future coming devices

This is the case only with poor implementations that don’t have sensible defaults.

> Responsive design helps to create Web sites that are adjusting for different screen sizes.

True, and that is very useful but responsive design doesn’t help web sites that need to support a wide range of devices, going all the way down to low-end feature phones. User agent detect remains the only realistic way to publish to the full range of devices on the market, unless your site is very lightweight.

September 12th, 2013 at 05:49
1. Karl Dubost
  
  Hi Ronan,
  
  Thanks for these good questions.
  
  > RFC 1945 (HTTP 1.0, May 1996) and RFC 2616 (HTTP 1.1, June 1999) says the same thing.
  
  Agreed with that these two specs say. We learned since 1999 how painful it is. It’s why HTTP/1.1 bis is near completion (Last Call) and should be published very soon, it includes the prose I put in that post.
  
  >> sites will break just because they were not planned for the future coming devices
  > This is the case only with poor implementations that don’t have sensible defaults.
  
  We have to deal with reality unfortunately. The Web is a legacy code and content machine. Basically, it is about history, the fabric of time over content. Things get rusty and eventually break. We both agree about the sensible defaults. It’s the last point of my checklist “Always provide a working fallback whatever the solutions you chose are.”
  
  >> Responsive design helps to create Web sites that are adjusting for different screen sizes.
  > True, and that is very useful but responsive design doesn’t help web sites that need to support a wide range of devices, going all the way down to low-end feature phones. User agent detect remains the only realistic way to publish to the full range of devices on the market, unless your site is very lightweight.
  
  Yes. This is a very valid point and a more general question about the type of Web we are building. It has deep roots into the way you are creating responsive design for your Web site. That would deserve an article about the way people are using feature detections and responsive designs. To make it short, it is basically lightly brushed in the sentence “You could fall in the same traps as the ones existing with user agent detection algorithms.” Sometimes Responsive designs and/or feature detections are used to degrade the user experience for devices with lower capabilities. I usually prefer the other way around. A Web site should be working without JS or anything and still be readable on a low-end device. So basically, desiging for the lowest possible and improving with mediaqueries for bigger sizes and/or more capabilities. The low-end should be the norm (personal opinion). It doesn’t solve WAP low-end devices, and it’s why the checklist is proposed as a Zen practice and not as rules. For some low-end devices, there have been solutions for a long time: Opera Mini, UCWeb browser, some Nokia browsers, etc. do content adaption on the fly, aka the responsibility of the client and the device with regards to the browsing experience. These browsers have a lot easier task if your Web site is designed with the fallback being the minimal normal.
  
  Hope it clarifies.
  
  September 12th, 2013 at 06:22
Ronan

Thanks for the replies, Karl.

I think we’re mostly agreed on the important stuff—sites should default to delivering a low-end experience, and use whatever technique they wish to enrich this experience if the client supports it.

I agree in principal that progressive enhancement techniques and feature detection should be capable of yielding a solution that builds up gracefully from the lowest-end feature phone to smartphone/tablet but reality suggests that this so difficult to achieve in practice that the major players (Google, eBay, Yahoo Facebook et al) all opt to use user agent detection instead. You can see just how different an experience these brands serve up to different devices here: http://prism.mobiforge.com. I can’t think of a single website that supports the full range of web clients (feature phone -> smartphone -> tablet -> desktop -> TV) that has managed to make progressive enhancement work.

On the getting rusty & breakage point, yes, user agent detection requires updates and maintenance (typically outsourced to a service) but so too do countless other aspects of their web publishing infrastructure. Without constant maintenance none of these major players’ web infrastructures will stay up and running for too long (days even?). It’s not like it’s a bunch of static HTML sitting on server somewhere so it doesn’t feel correct to say that user agent detection is a more significant weakness in the future operation of these systems than anything else is.

September 12th, 2013 at 07:03
Ivan Dejanovic

Ronan, Karl first of thanks for insightful comments. I just want to ask some very practical questions.

First how low do you go when in supporting low-end devices? It is really not a naive question because there are still legacy browsers that don’t support a lot of features user in modern web development. How can a developer like me realistically determine what is the low-end experience users can reliably receive?

Second responsive design is all fine and dandy don’t get me wrong I like it a lot. I am implementing it in my products with a mixed degree of success. But opening nba’s official site for instance on my Nexus 4 is showing me a completely different site especially designed for mobile devices. Or Mozilla add-on site. Responsive design is hard to achieve if your web site or web application is not simple one page demo. Like Ronan said big names on the web, Mozilla being one of them, are not using responsive design for all their products. With that being the case what real chance I have as a single developer working on my suite of web apps to make them all work across multiple devices while maintain the richest user experience possible on every device my users have when companies with vastly superior development resources opted not to use responsive design.

To try to finish my comment on a positive note. I will continue using responsive design because I believe that is the right way to go and I am not frightened by big players giving up on that idea for some of their products. I don’t give up easy :).

Thanks for the great article

September 13th, 2013 at 02:15
1. Will Mitchell
  
  Hi Ivan,
  
  I’m also a single developer making my own page. I have the same question as your first one, but I have a different perspective on the second one. For me, since I certainly don’t have the time or resources to track down many devices for testing, responsive design is the only way to go. Bigger companies may have the developers to throw at detecting multiple devices and giving them individual treatment, but I certainly don’t. So I put in the extra effort at the start and bank on the fact that I can test a responsive design on my laptop, tablet, and phone and the gaps will be okay, too. Plus, the maintenance savings in the end are ridiculous compared to user agent string detection.
  
  Anyways, just my two cents. Great article, Karl. Thank you.
  
  September 13th, 2013 at 07:35
  1. Karl Dubost
    
    @Will,
    
    yes testing for many devices maintaining algorithms and logic is very resource intensive… and it is breaking stuff at large. :) And it’s not an imaginary issue. Enjoy http://mzl.la/1daYnoR ;)
    
    Thanks a lot for your comment.
    
    September 13th, 2013 at 08:48
2. Karl Dubost
  
  @Ivan,
  
  Could you define what is a low-end device? It’s not a rhetorical question. I think most people have different assumptions on what is the normal base depending on our own environment context. A low end device in a rural area of Africa might not be the same of a what is called a low-end device on the 101 motorway in the silicon valley. Which creates plenty of secondary issues.
  
  So if we are talking of browsers with a less advanced feature set, then we have techniques here. As I said previously, Design for the less advanced and improve with feature detection instead of user agent. It will not be perfect, but it’s more about having a practice which upgrades itself to a more advanced environment than trying to downgrade the experience.
  
  hehe hear hear about the NBA. We are in the process of dealing with representatives of the Web site. They currently do UA sniffing and that creates issues for Firefox OS because it was not in the list of user agents which were detected. It is the perfect example of why UA sniffing is not really working.
  It’s here https://bugzilla.mozilla.org/show_bug.cgi?id=843154
  
  We are lucky because in the case of the NBA, we have very nice responsive people on their team helping us to fix the issue and do the appropriate sniffing. From what I understood they have at least 3 sites: Desktop, mobile WAP and mobile Touch.
  
  Now if you look very carefully at these 3 sites, they do not offer the same content and the same information (not talking about the layout). It’s a topic I have not addressed in this article. Very often redirection based on user agent detection is not used to show the same content with the markup and the layout tailored for mobile devices. It is a different site. As a matter of facts, I often used for some sites the mobile domain name on my *laptop* because I want this type of content. It’s a different application. It then becomes very annoying when the site forbids me by automatic redirection to access one or the other site based on my environment. It’s not anymore a technical issue. It is a UX and a business decision. Fortunately some sites will allow you to go from one site to the other.
  
  So for responsive design, we should really separate the notion of type of Web sites with the notion of layouts. People are often mixing the two.
  
  And I didn’t think you were negative in your comments. You made very good points. All we do is part of an ecosystem where our techniques evolve at the same time than the technology. We will have better solutions in the future. This will change. I have seen that in my last 20 years of doing the Web ;)
  
  September 13th, 2013 at 08:43
3. Karl Dubost
  
  @Ivan,
  
  another good article when Responsive design go rogue and why it is important to design with low capabilities in mind.
  http://stephanierieger.com/a-plea-for-progressive-enhancement/
  
  September 13th, 2013 at 12:04
4. Bridget
  
  > How can a developer like me realistically determine what is the low-end experience users can reliably receive?
  
  Start with solid, well formed HTML. The web started with this portion…and works well all by itself, even if it isn’t quite slick and sexy. Things only start to “break” when we (heavy handedly) introduce complex CSS and JS.
  
  That is why utilizing fault tolerance in intelligent ways is great. Sticking with a progressive enhancement philosophy to layer on more for browsers that can take advantage of more can be a sanity saver, too. :)
  
  September 15th, 2013 at 12:24
RalfMC

Correct me if I’m wrong, but does the following not mean EXACTLY that tailoring responses to specific user agent strings is an intended use?

“If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used.”

September 13th, 2013 at 07:59
1. Karl Dubost
  
  Hi Ralf,
  
  The specification here is acknowledging another part of the daily business reality. There are different scenarios where users fake the user agent string of the client. A few of them below.
  
  * Web Compatibility
  * Bots
  * Web testing
  * Bypassing firewalls
  
  All of these use cases express the *user intent*, not the server intent. What is happening with user agent detection and tailored response from the server is that very often they make an assumption for the user. :) It will be fine plenty of time, and indeed it helps create services directed to one person. The issue is not when UA detection is successful (we would not be talking about it ;) ), but all the time when it is not successful. And this is happening very often.
  
  About UA masquerading not chosen by the user, Opera has the browserJS and Firefox the UA override, with a full cascade of consequences. Basically it is because servers are assuming a lot about UA strings that clients had to start to lie about what they really are. Fully by faking totally the user agent, partially by things like:
  
  Mozilla/5.0 (iPad; CPU OS 6_0_2 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A550 Safari/8536.25
  
  * Mozilla/5.0 – No it’s not :)
  * KHTML – No it’s not, though it’s your father ;)
  * like Gecko – No, really that’s not the case. Stop lying to me.
  
  You see the rabbit hole we are going into. I have a file right now on my drive with… 386 844 unique User Agent strings.
  
  September 13th, 2013 at 11:58
Ivan Dejanovic

Karl thank you for your response. Sorry if I am resurrecting and old thread but I find this subject very important and I am still not happy with my knowledge on this subject.

Take my comment below simply as an example for low-end device discussion not something I would actually do or propose.

When I started college back in 1997, yes I am that old :), we had a small computer lab students could you. Since number of computers were limited first year students had only some UNIX terminals at their disposal. We had to use some program similar to UNIX less to browse the web. It could only display HTML in the terminal and had some basic key commands for navigation.

If we take this as our starting point for support and use feature detection to build upon it we could get a good result and support wide range of devices. But how practical would this approach be? How realistically is that we could find a user that would use our system on such outdated device?

Maybe more realistic approach would probably be to set our starting point a bit higher. By doing this we would not lose many users, if any, but would save developers from much, probably unnecessary, work.

Again I apologize if I am a bit negative, or boring, but I find this topic very important. After years of desktop web development at my company where my requests always were make it work in IE7 and above on screen size 1024X768 and above I am trying to develop some open source web apps that might end up on Firefox OS. I am in completely different waters doing stuff in may spare time, trying to learn how to make my apps as good as possible on wide range of devices, most much weaker that my six core 8GB desktop or Nexus 4 mobile, while trying to keep the work load low so I manage to complete my apps in a reasonable time.

Thanks for all your help and thanks for the article links I have enjoyed reading them.

September 16th, 2013 at 02:53
1. Karl Dubost
  
  @Ivan
  
  Hehe. in 1997, I was working in a Web agency for two years already. So nah you are not that old ;)
  
  Very short answer. Design for the Web not for software or devices. :) It seems to be an easy answer, it is not. There will always be edge cases and there will always be compromises. The catastrophic UA detection we experience on a daily basis is often not visible for users who are deep into the technology, because they have a tendency to use the last shiny toys and software.
  
  The Web as I said earlier is a legacy machine. We createD sites in the past by targeting specific capabilities, such as a screen size or specific JS. If the site is the not working now on newer devices because of scripting and/or screen sizes, it exactly shows how wrong we were. Now the tendency is to design for WebKit. In 10 years from now, who knows what will be the mess we have to deal with because of that. :)
  
  September 16th, 2013 at 03:59
Remi Grumeau

That’s pretty funny to read this on a Mozilla website, since you guys weren’t really against UA sniffing when it was all about sniffing out IE6 10 years ago ;)

Why still serving CSS2, png, or nojs fallback to an iPad?! It makes extra bandwidth consumption and feature detection tools like Modernizer are not making any sense for well know devices. Why asking an iPad or a Nexus4 if it supports svg, CSS gradients or which video codec it can play on every damn page load?! Are you asking your friend their name and what they do for a living everytime they come by?

I’m sorry, but only a server-side profiling mechanism based on UA sniffing is a winning bet for serious Responsive Web optimizations for today. I’m agree it’s long, complicated, on the long run, and need to be updated every month at least, but editing content for thousand devices can’t be an easy task: deal with it!

September 17th, 2013 at 14:08
1. Karl Dubost
  
  Bonjour Remi,
  
  Tout d’abord une petite erreur de casting. I have written the article and I’m an employee of Mozilla since last July, but that doesn’t make this article be Mozilla in any fashion, not more than one of the voices of the community. Before I was working for Opera… and before for a Web agency. Anyway :)
  
  > Why still serving CSS2, png, or nojs fallback to an iPad?!
  
  I’m not sure, I understand this one. but I suspect you are talking about the case where the design is not mobile first. I think we are entering in different type of philosophies with regards to Web design, content and user experience.
  
  The issues described in the article are not about giving a good experience to well known devices but about blocking the unknown devices. This is what is happening.
  
  As you say, it’s not only long and complicated, it fails all the time. I encourage you to come help us contact the Web sites which are doing it wrong. It’s a work open to anyone who wants to give a bit of time. ;) It’s usually a good training for your own projects. You might decide to not care about certain devices and browsers, but then again, here it is a different philosophy which is not the Web I want. :)
  
  Will you be at Paris Web 2013? If yes don’t forget to ping me. We can continue the discussion live. :) And nitpicking, hello-gurus displays nice on Firefox for Android, but fails on Firefox OS. ;)
  
  September 17th, 2013 at 14:53
Remi Grumeau

> User-Agent sniffing is a future fail strategy.
That’s what i was talking about. It refers to a mobile-first strategy backed w/ server-side profiling to generate optimized CSS & JS files.

That’s a very good example we have here: hello gurus website uses server-side profiling to optimize its weight. Works fine on 98% of browsers & devices, yet might fail on new players, aka FirefoxOS. Agree, this is where this mechanism needs constant work and updates. And sorry if that hurts a few here, but the web & real people using the web today are not using FirefoxOS, and don’t have a clue it even exists. So why should I pay my devs to adapt a website to a platform nobody uses? I’m not supporting (aka testing) Bada or WebOS.

But on the real world, a new platform or browser is not coming out from nowhere and be > 10% of the market share in a couple days. Google Chrome, w/ all $$$ spent in marketing did reach 30% in 4 years! Android 2.3 is more than 3yo and still used by most.

Like said previously, don’t bash the technology just ’cause some are misusing it (yes Google experiment I’m looking at you!!). UA sniffing is not a future fail strategy, it sadly is a necessary evil.

September 18th, 2013 at 00:20
1. Karl Dubost
  
  >But on the real world, a new platform or browser is not coming out from nowhere and be > 10% of the market share in a couple days. Google Chrome, w/ all $$$ spent in marketing did reach 30% in 4 years!
  
  What I was mentioning in my previous comment. Philosophy of Web dev. :) I don’t think we will resolve this one. But thanks for adding a different opinion.
  
  September 18th, 2013 at 03:23
Remi Grumeau

(for your pressure, i fixed hello gurus’ website detection so now B2G is detected as a mobile too (and not firefox desktop) so you get the mobile UX

September 18th, 2013 at 01:41
1. Karl Dubost
  
  Thanks!
  
  September 18th, 2013 at 06:10

Comments are closed for this article.

Hacks

By Robert Nyman [Editor emeritus]

History

User-Agent Detection

Firefox and Mobile

Summary: UA detection Checklist Zen

About Robert Nyman [Editor emeritus]

24 comments

User-Agent detection, history and checklist

By Robert Nyman [Editor emeritus]

History

User-Agent Detection

Firefox and Mobile

Summary: UA detection Checklist Zen

About Robert Nyman [Editor emeritus]

Discover great resources for web development

Thanks! Please check your inbox to confirm your subscription.