If you’ve been involved in the SEO industry for very long at all, you’ll undoubtedly be aware of the importance of robots.txt files in certain circumstances.
Robots.txt files can be very useful in determining how your site is viewed by Google and the other search engines out there. Not only can you forbid the search engines from seeing (and therefore indexing) your site at all; you can also hide specific pages from their view.

For bigger sites, such as ecommerce platforms with hundreds and hundreds of pages, this can mean configuring quite a sizeable robots.txt file. And this is exactly where the problem can occur. Via his Google + page, Googler John Mueller recently reminded SEOs that, ideally, robot.txt files should be no larger than 500kb in size.

According to JM, Google’s web spider, or Googlebot for short will only read the first 500kb of any robots file. This means that if you have important information in your file, but it’s larger than 500kb then you could be leading Google to disregard said information. This could lead to serious problems and could even mean that your robots.txt file doesn’t succeed in performing the task that you originally configured it for – potentially causing big problems with your SEO campaign.

More than this, an unnecessarily large robots file could be exactly that; unnecessary and could suggest that the entire build and coding of your site might need a rethink. With Google and SEO in mind, remember to check your robots file after any work is done on the site, and remember that the simpler your site is to navigate, the better chance Googlebot has of doing so successfully.

Google have announced a nice update to Webmaster Tools this week that I’m quite pleased about, and sometimes I wonder if Google follow the bad news, good news rule as this welcomed update comes after a potentially controversial algorithm change which Joel blogged about a few days ago.

Previously the sitemaps section in Webmaster Tools simply allowed you to submit an xml sitemap, and displayed the number of pages submitted alongside the number of pages indexed. That was it basically, and the fact that this feature had little data to get you excited about could misguide someone into thinking that xml sitemaps are not too important when it comes to SEO.

I certainly witnessed the power of sitemaps last week when a designer asked me to look at a new site they had just inherited which didn’t even rank for their own brand name – which was not keywordrich.com by any means. After a quick xml sitemap submission in Webmaster Tools, the site jumped straight up to position 4 on the first page of results within 2 days.

Content-Type Data

What’s really great about the latest version is that there is a dashboard which displays the data separated into content types – web pages, images, video and news. For each content type you can see the number submitted to Google, and the number indexed by Google – and even presents this information in a colourful bar chart – data-obsessed folks love charts right?

Sitemap Errors

Another nice thing about the new feature however, is the data in the table below the graph. Webmaster Tools now show the different sitemaps that you’ve submitted, and the number of errors or warnings for each one, if there are any. It helps you to see the types of issues better apparently, although I haven’t seen this for myself yet (phew!).

Test your Sitemaps

Probably the best thing about the update is the ability to test your sitemaps before you submit them. So you can upload one and test whether it has errors or warnings and then fix them before submitting it to Google. On their blog, Google say it only takes a few seconds for the test to come back with results which sounds good, but they’ve also stated that not all errors or warnings will show up in the test, so you should probably check Webmaster Tools after you’ve actually submitted your sitemap to make sure.

Now with Google’s latest update to the sitemaps feature, I think this is the attention xml sitemaps deserve. Oh, and it’s worth noting that the video sitemaps feature in Webmaster Tools is no longer, now that this data is shown in the main sitemap section.

What feature in Webmaster Tools would you like to see updated next?

If you’re a fan of Alanis Morissette, you might be interested to know that not that many things she actually sang about in that song were ironic, which is kind of annoying. But late last week I came across something that’s a little bit close to ironic; Google announcing that a recent algorithm change is designed to target and penalise ad heavy websites.

According to their blog, Google are targeting sites which have an extreme number of ads on their site – specifically though they’re talking about sites with too many ads above the fold. Their argument is that sites with lots of adverts above the fold make it difficult for users to find the useful content which Google has directed them too, thus affecting the all important user experience.

So will your site be affected? What type of sites will suffer a hit to their rankings?

Well, to get an idea of the kind of site this algorithm change might affect, imagine a site that places heavy prominence on ads above the fold. A site that optimises hundreds of pages so that each one presents ads before the more natural, useful content, and which does this to monetize and bring in as much revenue as possible. In fact, imagine Google.co.uk.

Any one of Google’s own millions of search results pages could be considered offenders under this new algorithm change, simply because of the number of ads in place. I’ve just run a general search and the resulting page has 7 ads above the fold and three organic listings.

My point is that although Google’s search results pages have lots of ads above the fold, (for now) these ads are relevant and useful, and don’t detract from helping me find what I’m looking for.

This brings us to the crux of this new change – how much is too much? Google might argue that their 7 ads above the fold are acceptable – but then how many is too many in terms of other sites?

It’s also ironic that Google have announced this algorithm change when they own and operate one of the web’s most popular ad display networks. Does this change mean that Google will begin to penalise organically those sites which use its own ad display network too religiously? Perhaps Google haven’t noticed this irony yet, or perhaps there’s provision in the algorithm which effectively ignores Google’s own display network but penalises others. In theory, this would certain help to solidify the display network’s dominance as the choice of the masses, but there are issues there in terms of fair competition.

We’re unlikely to get an answer to any of this in the short term, but it’s worth bearing in mind next time you’re placing ads. Ominously, Google suggested that this change isn’t limited to ads above the fold either. In the blog in question they seem to suggest that if a site has only a small amount of content above the fold then it could be penalised in search, regardless of whether the rest of the above fold space is taken up by ads, flash, video or anything else.

Bear this in mind if your site’s take a hit in the coming weeks.

I think you’ll all agree that it’s been a very busy month so far! With everyone in SEO working at light-speed to get their work done, it’s a surprise there hasn’t been smoke coming from the keyboards….although that would have explained the fire drill earlier….

Despite the workload, there’s still been some outstanding success within the department, so read on to hear about those who deserve a little recognition this month!

Kieran The Conqueror Masters The Blog Network

This week has been another busy one with another of our SEO stars adding a string to their bow. Special mention has to go to Kieran for his constant drive to learn new skills; at the moment he is getting his head around the ins and outs of PHP whilst also spending time building the new network of Creare Blogs, which has proved invaluable to the content crew. Many thanks Kieran, keep up the good work!

Page One, Position One – Just Another Day At The Office For Andy Allen!

SEO Ace Andy Allen, has been busy working his magic as always. This week he has been planting the seeds for success – sending London-based client Germinate Design to position one on Google with their key phrase ‘Garden Designer London’.
Also reaping the rewards of Andy’s labour this week are Edwards Davies – who have seen their phrase ‘Personal Injury Solicitors London’ reach number one in the Google rankings – a great achievement considering the volume of competition out there.

Highlight from the Highlands

New clients are what we thrive on here at Creare and we love working with forward thinking companies from across the United Kingdom. For Hannah Gibson, this happened when gaining a client from all the way up in Scotland. Trust Deed Scotland provides ‘Life after Debt’ and being a fellow Scot, Hannah is particularly excited about building a strong relationship with such an innovative company in the coming weeks and months.

A big thanks to all of this month’s contributors, and we look forward to hearing your SEO stories for February!

Around a week ago, Google unveiled the next stage of their personal search integration: ‘Search Plus Your World’.

Basically, before I go into details concerning the feather ruffling said announcement has made, let’s take a quick look at what this new initiative involves.

1. Google + Photos & Posts in Search

Now, when you’re signed into your Google + profile Google will begin to push Google plus photos and posts from your friends directly into your search results. Personally, I can’t think of anything more annoying; if I wanted to search Google + I would go on Google + but there you have it, it’s done.

1. Profiles in Search

Following the update, Google now also pushes Google + profiles into search results. On their blog, they state that they’ll only suggest ‘people you’re close to or might be interested in following’. You might find this useful if you’re logged into a business account as an SEO or marketer, but do you want networking suggestions when you’re simply doing some online shopping? Again, it’s unclear why this has been integrated into web search and not left in the ‘Suggestions’ area of Google + where most people will happily go to find it.

1. Pages

Finally, Google is also integrating Google + pages into the normal web search results. The idea is that if you’re searching on a topic you might be interested in joining groups and pages related to that topic. While this might be the case, again there’s an argument for keeping such things separate. If I wanted to join a social media group on knitting I’d search Facebook or Google + directly for knitting groups.

And this is the major problem that I forsee with these changes. The reason many of us use Google web search is because we want to find just that: web pages. Most modern day internet users are savvy enough to know where to go to find social media groups or profiles and simply want old fashioned web pages returned for their regular searches.

Arguably, it was Google’s fresh and simple design which contributed to its early successes, not over complication. With the introduction of these Google + features into search, I fear we’re one step closer to over population in the search. We already have rampant adds, shopping and image results, reviews, +1 histories and much more crammed into one page and I wonder if Google is moving toward creating a restaurant menu which simply has far too many dishes on it to choose from.

Google have shown time and again that they’re committed to returning the most useful and relevant results for users, which is an admirable goal. But arguably, by cramming the SERPs with data from all kinds of streams (images, social media posts, news, product items etc) Google is making the SERPs less relevant, as at least 90% of the results returned will be the wrong stream for which the searcher is looking. For example, if I’m looking for new shoes, neither images of shows, a site about shoes, or a group about shows is going to interest me.

So this brings me to my final point. The Search Plus Your World announcement has sparked even more calls for Google to be investigated for its conduct regarding competitors. You’ll notice that I didn’t mention that the search giant was integrating Facebook or Twitter profiles into its search results did I? That’s because they’re not, and this is the major problem. Many argue that Google are unfairly favouring Google + with this update and are therefore abusing their market dominance.

Time will tell, but in the meantime, whether you’re a fan of the changes or not, it’s worth noting that, as an SEO consultant, the success or failure of these changes is important. If they succeed in making Google search more popular, then more people are going to be signing up for Google + profiles, and more than this, more people are going to be searching whilst signed into their profiles. This means that the keyword data that many SEO companies rely upon will continue to decline, as the ‘not set’ section of Google analytics grows.

If this new personalised search is less successful and is perceived as annoying by users, we might expect to see more data on keyword traffic to our sites, as people sign out in order to have more of a simple and traditional search experience.