In essence, the gist behind a robots.txt file is simple: Also known as the Robots Exclusion Protocol, or robots.txt protocol, the file is a standard, nothing-special text file which has a certain format to it and is used to prevent (willing) web spiders and other web robots from accessing all or part of a website which would otherwise be open to the general public.

A simple example

Say you’re running a competition site and you don’t want Google (or any other search engine, for that matter) to access the list of competitions you currently run which sits on a sub-directory called /comp-list/ [and bear with us on this one, it's just an example!]. All you’d have to do is not allow any bots/spiders to crawl that page, or more importantly, anything in that folder by disallowing access to it. The way you do that could not be simpler; all you do is create a robots.txt file, which will sit in your main root directory, with the following information:

User-agent: Name-of-bot-or spider, i.e. Googlebot
Disallow: /path

In the above example, ‘User-agent’, simply dictates who it is you are targeting with this rule, whilst ‘Disallow’ is a special keyword which becomes an action of that target. So putting that all together, your robots.txt file should look something like this:

User-agent: Googlebot
Disallow: /comp-list

Or, to make it so that no robots/spiders can crawl the folder, simply replace Googlebot, below with an asterisk, or “*” which is in programming terms a visual representation of “all”.

User-agent: *
Disallow: /comp-list

That’s pretty much it! Well, if all you wanted to do was hide that page/folder that would pretty much be it. However, herein is the rub — What most people tend to forget is that, just because you stop Google or other engines from accessing your site, (and most common search engines inc. MSN Live, Google and Yahoo! will listen to what your robots.txt file says) – it does not stop people from opening your robots.txt file and finding our what you’re trying to hide from the world.

That’s pretty important to realise, especially when you are putting away invoices, or anything else to good use in a folder called ‘/secret-stash’. Google and other bots might not post it for the world to see, but it doesn’t mean anyone who sees your file can’t find it! So be weary of that when you are working on your robots file. The question that begs to be answered is why you’d put your invoices in a folder called ‘/secret-stash’, online in the first place, obviously : ).

Another important thing to remember is that robots.txt does not guarantee privacy! It works on the simple basis of search engine cooperation; having said that however, the big players in the search game at this stage do support it!

An intriguing scenario

One thing that never ceased to amaze me was how ‘thorough’ the Whitehouse’s official robots.txt file was- and we stress ‘was’. After eight years in power, President Bush and his web boffins had racked up nearly 2400 lines worth of search engine exclusions in the robots.txt file, which include:

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
...

And even more… Now some would say there’s a big conspiracy going on with political staff trying to hide information to the public about 9/11 or results, or most importantly, “expectmore” – which is sort of ironic, considering you are getting less information? No? Anyhow, We’ve recently taken a look at the new Obamafied version of the same document and it now looks a lot leaner; in fact, very much leaner… This much leaner:

User-agent: *
Disallow: /includes/
Disallow: /search/
Disallow: /omb/search/

However, on a side note, the last two entries were recently added in addendum to ‘/includes/’. Interesting to say the least right? I mean it may seem like a stupid/childish topic to even get into, but there is a method to the madness of this discussion.

We wondered if this showed a changing nature of how the world- or at least ‘The Whitehouse’ is viewing the whole process of information gathering, retrieval and dissemination –  or more importantly, perhaps trying to be more open? And then I found this article on BBC which went something along the lines of:

“Searching for data about the Obama administration should get easier as the Whitehouse.gov website gets overhauled.
Barack Obama’s new media team is letting search engines index almost everything on the site.
By contrast, after eight years of government the Bush administration was stopping huge swathes of data from being searchable.
The move is part of President Obama’s larger push to make the US government more open and transparent.“

And then it hit home! Maybe there really is a change coming? Point is, at least the team at Obama headquarters knows how these things work and believe me that’s a good start!  Puts a whole new spin to ‘The Freedom of Information Act’.

Either way, maybe it’s an important strategy, or perhaps it’s just Obama’s team weeding out any last remains of the old administrations’ shortcomings? What it does show is that government, whether locally or overseas is realising the importance the web plays in our daily lives, especially in accessing information, and is taking every step they possibly can to rectify any mistakes which may have caused a roadblock of sorts in the past. And in anyone’s books, that can’t be a bad thing.

Read more about search engine optimization and robots.txt procedures or check out The Web Robots page for more information on robots.txt.

To kick off the new year in Web Design, I thought it would be best to go through a couple of my tell-tale signs that can help you gauge if and when your website’s design sucks. Obviously, “Website design” in this concept does not only include the graphical elements of the page, but of course the Information architecture, usability, accessibility, etc. So without further ado– Let’s get going with round one of “Your website design sucks because…” with Your website design sucks because users can’t find crap!

1. What the heck are you thinking?!

Whether or not you are running a corporate information site or a 15,000 page e-Commerce store- your content should be easy to find! The biggest problem is people think that you ALWAYS need a search box. Search on a website comes in many flavours, and sure; a search box would be awesome- as long as the results are easy to understand and filter to the right place. Having said that, not all great websites have search boxes or website search for that matter – but most if not all of them [great websites, that is] make the user experience worthwhile by making their content easy to find. Here are a few tips:

• If you are running a blog, make sure there is at least a way for the user to find an archive of all your content. After all, one of the main calling-cards, if you like – of a blog, is the fact that it is a chronological listing of posts or entries. Other important elements that help users searching for content is Tags and Categories. Try to make sure your posts only belong to one parent category. I.e. Not “Sydney Web Design” AND “Web Design” — after all, 99% of the time, one of the two (or more) categories makes the most sense. Tags help users by allowing to search for other posts which they might find interesting based on a common interest; i.e. Usability.
• If you are running a website; other than a web log , make sure that there is a common, geographical structure to your pages. I.e. there should always be the same header and footer on all pages – including any side-navigation or side elements. Doing this makes sure that users learn where things are and helps elevate any problems in looking for things all over your website.

2. Some conventional wisdom

Most webmasters and/or designers usually get search from a UI point of view under control. I mean come on! It’s basically three elements– A label “search for:”, a text box, usually pre-filled with “type query here” or something of the sort, and finally a “go/search/find” submit button… Easy, right? Wrong! Whilst the front-end features of site search are seen as simple, most websites/intranets (even the big players) crash and burn on the search results page.

3. A case study in Search Usability: Australia Post

Ahh, good old Australia Post! I love the fact that they deliver to anywhere in the world and the prices are not too bad — and I haven’t had anything major get lost whilst sending out *so far*. However, one thing that drives me insane about Australia Post is their website; http://www.austpost.com.au — not only is this website in need of a “major” overhaul. Let’s start with the essentials of this particular topic; Search.

The search on AustPost is located at: http://search.auspost.com.au. Nothing wrong with that so far- In fact, it’s smart to have search on either a sub-domain or a sub-folder at search.domain.com or domain.com/search. Moving right along. Oh wait! That domain does not work (Under Construction!!) unless you append the following to it: “/cse/auspost/” making the FULL-URI http://search.auspost.com.au/cse/auspost/ – Tsk, tsk AP!

If you type “test” in the search box and click the search button of the main page you get diverted to a search results page; which is okay so far– However, the diverted results page is a TOTALLY DIFFERENT website (in terms of overall look and feel). Check it out for yourself.

Australia Post Search Box - Main Page

All good so far… Let’s see what happens when we get diverted to the ‘actual’ search page.

Australia Post Search Results Page - Search.austpost.com.au

Oops! Something went completely wrong. Did I, as the user kill the site? Or is it meant to be a completely different site followed by a completely different user experience? [Both valid questions you don't need a visitor to your website, especially your search page asking!].

As Homer Simpson would say in a time of crisis; Doh! Different Logo, Different colour scheme, Different tab system, Different IA… Well, at least the logo is generally in the same place as before. Granted, I don’t know the reasoning behind this– Maybe they are in the process of moving sites across to a newer(?) design – but nonetheless, I think being a government corporation, this is pretty bad usability!
Anyway, let’s move right along… As if that first problem wasn’t bad enough, let’s see the search results form. Below is a screenshot of the “general” search form. By general search form I’m referring to the non-advanced variety!

Search Results Box

Wow, okay — where to start, where to start. Let’s rattle off some of the issues with this search box/form.

1. Why have a “Show search options >>” as well as “‘Advanced Search”. That is not only confusing, but a waste of visual space/real estate. Why not make general search easy to use with no extra-ordinary features, and leave advanced search for those people that need to dig deeper into the data mine. Ahhh, I slowly started to realise why, and you will see why soon!
2. See the “Select All” and “Clear All” check boxes? They only select/deselect one option — and that is the “Australia Post Website” check box. WHAT WERE YOU THINKING AUSTPOST DESIGNERS/DEVELOPERS!!! Give me three good reasons why this is necessary on this page (in its current format) and I will stop using Facebook for a month… Okay, a week! Not only is this stupid from a usability/IA point of view, but it’s NOT needed here. Why?, you ask?
   1. When you actually decide to click on “Search” you really DON’T end up having a choice in the matter, because the “Australia Post Website” check box is auto-ticked when the page has reloaded with the search results! Oh Oh!
   2. It’s a waste of a control to begin with! The same thing could be easily achieved with one button which toggles through the selections. I.e. starts off as “Select all options” then changes to “De-select all options” upon click… However…
   3. There is NO need for the control at all since there is NO other option/area to search from. I.e. If there was “Australia Post Website” and “Australia Post PO Box Search”, etc., then it would make sense — however, there is no other option!
3. The “Exact Phrase” check box control is USELESS in all formats of the word! In theory, the “Exact phrase” should let you search for the term “test” instead of the term test. What this ultimately means is that in reality, “Exact Phrase” should only find the search query if it exists EXACTLY in that format, and not part of other words, or as part of a phrase, etc. The only problem with this is that selecting “Exact Phrase” does NOT change the result set; in essence, returning the SAME number of results as well as in the same order! Yay for happy searching — Not!
4. Clicking “Show search options >>” reveals two fields/controls which are apparently meant to help the user whilst searching but not be enough to warrant the name “Advanced Search”. When you click this link, two new fields pop down; a) Format, and b) Modified. Fair enough, you say. They are letting the user choose what type of file format to search; i.e. PDF, Word, etc. as well as the ‘age’ of the document(s) in question or when they were last updated. So what’s the problem here? Let’s take a peek!


Search Results - Modified drop-down

Wow! That’s probably the best word to describe this drop-down. Sure, Australia Post must feel great giving thier users so many options to choose from, but on what planet would you be from if you need THAT many options in sorting last modified periods? Surely, a) “Anytime”, b) Past 24 hours, c) Past week, d) Past month, and e) Past year – would suffice? No? Granularity of options is something worth your time as an Information architect or Website designer/developer investing in! The more options you give the user that they don’t necessarily need or will never use, the more they have to think. The experience should not make them think! Just give them results, and quick!

5. Don’t even get me started on the “Advanced Search” link! Let’s just say that it’s a little more advanced than what an “advanced search” needs to be. So much so that a PhD in Comp. Science would definitely help! I mean for Pete’s sake, there is a field called “Custom query” which lets you, quote “Create a query using search operators (and, or, not, near, quotes and parenthesis) and system fields.” unquote! WHAT THA?!?! Sure, I may get it, hell you might even get it as a developer or designer, but your average Joe who might need just that little bit more input into his or her search would most likely freak out! Take it off PLEASE! I’m not searching the CIA Central Repository here, people!

3. In conclusion…

Okay, so I must admit at this point in time that I’m getting a little bit more critical and more importantly a touch “harsher” in my blogging and reviews — but I think it’s important to look at problems like these that normal web browsers would AND DO encounter on a day-to-day basis surfing the Interweb! So hopefully you have picked up a few pointers from this blog post that will help you with your upcoming search page design/re-design. Remember, these are above all just some ideas of what not to do, but the field is huge! Make sure you get people with limited Internet experience to trial all software you create (if possible!) but more importantly, keep in mind some of these ideas for next time and I’m sure your users will thank you!

Thanks for comin’ back guys. Please leave your comments or critisisms below – and be sure to Subscribe to the RSS feed so you can keep up-to-date on all the latest goss and posts from Elastique Web Design Blog. Till next time, Cheb.