User Agent: Find Out Your Web Browser’s UA Here

Your User Agent is:

CCBot/2.0 (https://commoncrawl.org/faq/)

Your IP Address is:

US flag 54.227.31.145

Browser Information:

JavaScript Enabled:No
Cookies Enabled:No
Device Pixel Ratio:N/A
Screen Resolution:N/A
Browser Window Size:N/A
Local Time:N/A
Time Zone:N/A

User agents are unique to every visitor on the web. They reveal a catalogue of technical data about the device and software that the visitor is using. Armed with this information, you can develop richer and more dynamic websites that deliver different experiences based on the user agent that’s visiting.

User agents are also critical in controlling search engine robots using the robots.txt file on your server. But they dont function perfectly in every situation, and the information in a user agent string can be faked.

In order to leverage this information, you need to understand the component parts of a user agent string, and consider also the potential risks of using this method to deliver content.

What is a User Agent?

Everyone that is browsing the web right now has a user agent. It’s the software that acts as the bridge between you, the user, and the internet. It’s easiest to understand user agents if we backtrack and look at the evolution of the web, so we can understand the benefits of user agents.

When the internet was a text-based system, right back at the beginning of its use, users had to type commands to navigate and send messages. Now, we have browsers to do that for us. We simply point and click, and the browser is acting as our “agent,” turning our actions into commands.

When your browser (or similar device) loads a website, it identifies itself as an agent when it retrieves the content you’ve requested. Along with that user agent identification, the browser sends a host of information about the device and network that it’s on. This is a really set of data for web developers, since it allows them to customize the experience depending on the user agent that’s loaded the page.

User Agent Types

Browsers are a straightforward example of a user agent, but other tools can act as agents. Crucially, not all user agents are controlled or instructed by humans, in real time. Search engine crawlers are a good example of a user agent that is (largely) automated — a robot that trawls the web without a user at the helm.

Here’s a list of some of the user agents you’ll encounter:

Reading HTTP User Agent Strings

Once the user agent has identified itself to the web server, a process called content negotiation can begin. This allows the website to serve different versions of itself, based on the user agent string. The agent passes its ID card over to the server, and the server then negotiates a combination of suitable files, scripts, and media.

In the early days of the web, user agents were used to distinguish Mosaic from Mozilla, since Mosaic did not support frames, while Mozilla did.

To look at a user agent string in more detail, take a look at this example user agent string, as generated by the WhoIsHostingThis User Agent Tool. Your results will be unique to your computer, device and network, but here one from a computer we have in the office:

User Agent Tool Content Output

Breaking the example down, we get the following information:

Dissecting user agent strings can be tricky, since there is no standard format. But there are guides and analytics tools all over the web that can help. For most designers, the application, version and engine are likely to be key.

Note that a huge part of the user agent string is concerned with compatibility. That’s because Internet Explorer originally had to declare itself to be Mozilla compatible in order to receive content with frames.

In practice, the majority of browsers now declare themselves to be Mozilla compatible to ensure that they can access all of the content on the web.

Content Negotiation

So: the user agent string is a little muddled. But it’s still useful. What can we do with it?

We can:

Overall, we can empower our scripts to make the best choice for our visitor, based on their user agent. And we can feed that data back into a cycle of continuous improvement, analytics and other processes, like conversion optimization.

User Agents and Robots.txt

The robots.txt file is a file on your web server that controls how some user agents behave. In the majority of cases, we use robots.txt to tell search engine crawlers — or “robots” — what to do.

As we mentioned in the introduction, search engine crawlers are a very specific type of user agent. The information in the robots.txt file applies only to crawlers, and it’s up to the crawlers to interpret them as we intend.

Let’s look at some examples.

If we wanted to ban all crawlers from visiting a website, we’d create a text file called robots.txt, place it in the top level (web accessible) directory on our server, and add the following text:

Free country flag icons