Steve Johnston
The Google Blog of a Google Consultant

Google tapping into the invisible web: One of the most exciting Google announcements in recent months has to be how it is going about trying to find some of the hidden web. Google's determination to index the world's information sees it starting to submit data into web site forms and execute 'submits'. For many sites the visibility of content that is only found through keyword search is highly limited, if not completely hidden. Google has announced that it has started a beta trial where it will start automatically submitting search terms into such forms, with the hope that it will generate results pages that will lead to URLs of content it has yet to index. Apparently the source of the terms it submits will initially be the pages from the site on which it finds the form. Also it is only going to try this initially on 'high-quality' sites, which presumably means ones with a great reputation and authority, as measured by the quality of links they enjoy.

Hear me speak at SES London.

Advanced Track - Organic Listings Forum - 19th Feb 2008 - 9.30am

"Pose questions to our panel of experts about free "organic" listing issues, plus participate in this session that allows the audience to share tips, tools and techniques. There's no set agenda, so this is an ideal session to discuss any major recent changes with organic listings.

Moderator:

Mike Grehan, Co-chair, SES London and Founder and CEO, Searchvisible Ltd.

Speakers:

David Naylor, SEO, Bronco

Ralph Tegtmeier, aka Fantomaster

Edward Cowell, Technical Director, Neutralize

Steve Johnston, Google Consultant"

Sorry about the short notice!

BBC Radio Wiltshire Google Love Interview. I had fun this morning talking about how Google users search for 'Wiltshire' related information with BBC Radio Wiltshire presenter Olly Dearden and managed to plug the fundraising Google Love Lecture too! I have YouTubed the audio of the interview which you can listen to below; it is only seven minutes long. The list of Wiltshire terms we discuss in the interview are below.

1. wiltshire county council
2. wiltshire times
3. wiltshire college
4. jobs in wiltshire
5. schools in wiltshire
6. wiltshire police
7. wiltshire farm foods
8. wiltshire gazette & herald
9. bbc wiltshire
10. wiltshire news
11. wiltshire buses
12. wiltshire map
13. cottages in wiltshire
14. wiltshire libraries
15. holidays in wiltshire
16. wiltshire football league
17. this is wiltshire
18. wiltshire hotels
19. wiltshire estate agents
20. weddings in wiltshire
Oh, and Olly wanted to perform for the search term handsome radio presenter, so hopefully this link will give him a helping hand.

Update on Upcoming Fundraising Speaking Event: There are tickets still available. The old item page on eBay expired and sadly the relisting has a different item number (I really should pay attention sometimes). So for the relisted Google Love Lecture ticket page - all proceeds going to the school my children attend - can be found at www.johnston.co.uk/googlelove (why, oh why, didn't I do that last time).

Upcoming Fundraising Speaking Event:
I am presenting a '50 Ways to Make Google Love Your Web Site' lecture on the 1st November 2007 as a fundraising effort for the secondary school my children attend - St Laurence School in Bradford-on-Avon, Wiltshire. It is an evening event at the school, from 7.00pm and therefore may prove a little awkward for you if you are not in the vicinity.

The event price is a low £45 and there are only 110 tickets available. All the profits go to the school's major projects, in particular its new library. All attendees get a free copy of the new edition of my ebook (22,000 words, 56 pages), worth £19.95.

For further details and to buy tickets, please visit the Ebay UK page for the event (Ebay item no. 140170041904).

Edition 2 of 50 Ways to Make Google Love Your Web Site: Hot off the virtual press is my thoroughly updated ebook on optimizing web sites for Google search. I confess it is a little overdue and all of my customers who have bought one since the 1st January 2007 will be sent a copy of the new one free of charge. I have kept the price the same as the outgoing edition.

While maintaining the overall structure, I have expanded the content extensively, in fact by almost 100%, going from 12,500 words to 22,000 words. There are around 10 new sections.

For further details and the purchase page: Google Optimization eBook

Google Consolidates Reputation Across Duplicate URLs! In the middle of September, Google's webmaster blog posted a piece of information that is News! The blog URL is under here (it's very long). The crux of the matter is these three sentences:

1. When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.
2. We select what we think is the "best" URL to represent the cluster in search results.
3. We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.

It is point three that is so interesting. Consolidation of the link popularity (or Reputation, as I like to call it) is not what we have been assuming they did. That's very clever, if its really happening. I don't recommend you stop worrying about this as an issue, though. Google can't get this right all the time, better for the problem not to be there to solve in the first place.

Quick post on forcing regional results in Google.com. In a recent post from Matt Cutts, he shows a way of making Google.com show results as if it were a regional Google, such as Google.co.uk: Add the &=uk parameter to the end of Google's URL. The following results within Google.com then show me as the third most important 'Johnston' to UK visitors (he he), but only twenty first globally (sigh):

http://www.google.com/search?q=johnston&gl=uk http://www.google.com/search?q=johnston

Change the 'gl' parameter to the country of your choice and Google.com will provide the necessary results.

Update: Further down the Matt Cutts blog page I was catching up with, was a reference to minty fresh indexing - that described instances where Google crawled and indexed content within minutes of it being published - so I thought I'd test it. Within 5 minutes of the above post being published the following result was visible in Google's SERPs:

You may have spotted from the screen grab a lack of a link to the 'Cached' version of the page, which clearly hasn't happened yet, which suggests it has pulled the content directly from Blogger. I have my blog set to send a ping to blogger, which is Google owned, and I believe from my recent SiteLinks status that I am now an official authority site which doubtless increases the frequency of crawling on my site. Don't expect this to happen to every site, but it is very cool.

We are finding an increase in the use of font replacement techniques, such as sIFR (scalable Inman Flash Replacement), being implemented by our clients' web agencies and internal developers. sIFR is implemented as a behaviour layer (i.e. laid over normal mark-up and styling), and allows a website to display fonts - via cleverly-sized Flash movies - that it could not normally use.

sIFR has been created with accessibility (and therefore search-engine friendliness) in mind. sIFR is designed so that if either Javascript or Flash are not present, it simply doesn't present the Flash content. This is intended to ensure that search engines, and assistive technologies such as screen readers (used by vision-impaired users to read web pages out loud), are only provided with plain HTML content. Google certainly processes Flash, much as it processes PDFs, by examining the files and drawing out the text content. It does not rate such text as highly as it rates plain HTML, as there is little semantic information contained within such files at present, certainly far less than is provided by HTML text.

(Note there appear to be security issues with sIFR2 and 3)

There are several key questions:

1. Does use of sIFR prevent Google from indexing the replaced content? No. This has been confirmed by Google engineers.
2. Does use of sIFR trigger a negative signal to Google's cloaking detection algorithm? Yes, we believe so, albeit not one large enough to cause big problems on its own. Google engineer Mark Berghausen comments: "If you do have a reason to believe that your rankings in Google Search are being lowered because of your use of sIFR, I'd definitely encourage you to file a Reconsideration Request at Google Webmaster Central...". Mark's comment implies that although sIFR will not prevent the replaced content from being indexed, there is a risk there.
3. Does use of sIFR to replace a heading decrease the value Google gives that heading? Unclear. The risk is, we believe, low. The hazard, however, is great. Headings are extremely important content on any page and their devaluation through use of flash replacement techniques would be problematic, to say the least. Google could reasonably devalue any content referenced in a script for replacement until it's sure of what it is... maybe Google will make this assessment, but the only sure result will be a big flag for hidden content. Such content runs the risk of simply being ignored. Google refers, in the Webmaster Blog link post in point 1., to their hard and fast rule that the content in the HTML and the replacement must be identical; to check this algorithmically is challenging so they are likely to err against the hidden content until it is confirmed.

In the end, it is a question of a balancing the aesthetic benefit versus the risk of a depressed ranking. If you decide to take the risk, you may wish to consider launching your content with the replacement in place, and allow Google performance to stabilise. Then remove it, and see if Google performance improves.

As a final note, because of the potential use of such object-replacement techniques for spamming, it is something that will always need to be monitored. Should Google performance deteriorate, sIFR, and any other replacement techniques, would always be candidates for the investigation. We won't be using any such techniques on our sites, nor will we encourage our clients to do so.

This post was contributed by one of my lead consultants, Liam McGee. Thanks Liam.

Google & The Long Tail of Search: To help people comprehend how long the tail of search is, I have often used a reference that I remember reading but had long lost being able to cite directly; namely '50% of the search terms typed into Google are so unique they will never be typed in again'. Today, however, I am moving on from this quote to a new one that I can cite. Udi Manber, VP of Google Engineering said at a recent conference, that there are three reasons why Search is only going to get harder in the future. One of these I already use - "scale and diversity are almost beyond comprehension" - one of which is not particularly relevant to this point, but the third will replace my previous reference: "20 to 25% of the queries we see today, we have never seen before". I will convert this into '1 in 4 of the expressions typed into Google today have never been seen by Google before'. Ponder that for a moment. Google is tapped into our collective consciousness. It's astonishing.

Feel like I have really arrived today. Google has added Sitelinks to my site when searching for me as a Google Consultant. It may not be happening on all datacentres yet, so there is a screen grab below. My current understanding of why and how these links appear is as follows: The site needs to be perceived as an authority for the search terms the Sitelinks appear for - i.e. enough high-reputation and relevant inbound links - (I don't get Sitelinks when you search for Steve Johnston only for Google Consultant); Onsite link patterns and anchor text combined with off site link patterns and anchor text suggest the key pages and the labels to use; Users' search refinement patterns and site visits - as tracked by Google - help qualify user needs and the prioritisation of the pages in the Sitelinks list.

Deep breath. Finally. Google today allows you to properly interrogate your back links, via the Google Webmaster Console. One qualifier from Mr Cutts, "The backlink tool doesn't show 100% of the backlinks from Google yet, but I expect the number of links that are available to grow". Export functionality allows you to take it away to de-duplicate it. Of course you need to register your sites with the webmaster tool kit, but hey, its progress, with a capital P.

Couple of quick updates: The previous post has gone through the wash and the situation has stayed the same, which is interesting. In the meantime, the Wikipedia entry for a 'Google Consultant' has been redirected to the SEO page, thanks to rather too much attention, which is a shame.

Also, I have updated my http header-viewer, with a little help from my friends, so that it now follows any server redirects it encounters. I have used Rex Swain's version for years, for this functionality, so figured I should give his server a rest as I have never clicked on one of his adverts; sorry Rex.

Happy New Year. As the SERPs for a 'Google Consultant' hot up, one of my resolutions is to work diligently over the coming months to retain my top slot, and to do what I can to appear even more relevant to those looking for what I do. For example, the marvellous E-consultancy.com have recently added me to their supplier directory, which at the time of writing is on the second page of results in Google for the 'Google Consultant' search. So if I gently apply a bit of extra relevance to my entry in the directory, by pointing it out to you with a nice link: Steve Johnston Google Consultant in E-consultancy it should bump up to the first page. Now E-consultancy's PR7 couldn't possibly unseat me from the top slot... Or could it? Watch this space.

Plug: A quick plug for my Designer Jewellery client Astley Clarke. I am delighted at how hard they have worked to implement my recommendations for improving Google's understanding of what they are selling. Go and have a look, for yourselves. Incidentally, their jewellery makes rather spectacular Christmas gifts.

Link bait. Hmmm, such an ugly term, but a very real need at the moment. As the results pages of Google become saturated for most competitive online markets, the off-page factors, i.e. everything that happens elsewhere on the web regarding your site, become more and more important. I frequently make this point with my clients by saying that Google effectively cares more about what the rest of the web says about your site than what you do. Because of this, the development of content and initiatives that improve the likelihood of your site acquiring relevant links (the relevance is everything; we are not interested in quantity, but quality here) becomes one of your biggest priorities.

Examples are pretty easy to find: One of my counterparts in the US is Jaimie Sirovich, who is far too young to be a proper consultant, forgive me Jaimie, and he recently posted an entertaining piece of geek content as link bait. He writes about SEO from a very technical perspective, so acquiring geek links is valuable to him. Apart from the entertainment value, you can guarantee the main reason for creating such content in the first place was to acquire links, and quickly.

The best place to start an exercise in link bait is to look at what you already do and see if you can repurpose some of it. For example, you can look out for an example from me in the coming weeks where, thanks to a telephony quirk, I receive telephone calls destined for Google in the UK. Despite the temptation, I am very honourable and pass the correct contact number on, but in the meantime, I am logging why they are trying to reach our favourite search engine. It makes fascinating reading and I will soon publish my market research summary of the top ten reasons people phone Google. Whilst this exercise is apparently unrelated to optimisation for Google, the relevance to 'Google' queries is something I want to improve on in general, whilst queries about results in Google may well end up featuring in the top ten anyway, so there will be some overlap.

For those of you wondering about Google's approval of such tactics, just remember what we hear from them all the time; 'Build sites for users, not for search engines'. What could be more for users than the sort of content that users just can't help but link to, it is sooo interesting. So, get your thinking hats on. Why should your sector of relevant sites link to you? Huh?

Yes that was me on episode 7 of the BBC's Dragons' Den last night pitching my StoryCode business. I didn't really expect them to broadcast the bit where I said I was a Google Consultant, but they did! Further details will be found on the StoryCode blog in the coming days.

Google Mini gets smaller and cheaper! UK pricing is now £1,295 for up to 50,000 documents, with a ceiling for the Mini of 300,000 documents for £4,000. This remains the most remarkable value... £1,295 for 50,000 documents, blimey! Go see for yourself: Google Mini UK.

Switch off Google.com redirection to your local domain. A quick tip to stop your attempt to reach Google.com from being hi-jacked by Google's geo-targetting. Typically Google will serve you the Google site that it considers most relevant to you, based on the IP address from which your request has come. This means that most UK users get redirected to Google.co.uk when they ask for Google.com. Next time you want to get to Google.com, use www.google.com/ncr. This will tell Google not to redirect you. Don't then assume, however, that you are then seeing results that a US user will see, because this won't be the case. Google will still interrogate your IP address and attempt to serve you more relevant results. This will be most noticable in the Adwords campaigns that Google serves. The www.google.com/ncr request sets your Google cookie to not redirect, so if you want your redirection back in future, delete your current Google cookie.

Just a quick note to let you all know that Google Analytics has dropped its 'waiting time' to open an account. Visit the GA blog for further details.

NOODP Success: Reference my blog post from July 25th below; Google re-crawled my home page on the 27th July and the Title tag in Google's results was replaced by the Title from my home page on the 29th July.

Site Migration Made Easy (301 Redirects revisited): I have bumped into a number of questions recently about how to migrate a site from one technology to another, or even from one domain to another, without massacring Google referrals. I wrote the following post to one of the email lists I contribute to and figured it might be a good time to mention it again here.

If your primary concern is to defend the traffic you already have from Google then the issue of rolling out a new technology platform for your site is a very simple one to manage:

Your existing site will have a number of locations (URLs) indexed in Google, a bunch of which will also appear on third-party sites around the web. As a consequence Google will return periodically to these pages to attempt to re-index them and in the meantime will present these locations to potential users as relevant locations on which to find information to support their search queries. Google also evaluates the third-party links, in terms of their subject matter and authority, and confers additional relevance to the pages they point at.

So, when you dump all the old locations of a site by re-publishing it under a completely new organisational structure, paths and file names, a number of things happen - if you do not act in advance:

1. Google attempts to reindex a location it has in its index and is presented with a 404 File Not Found. Shortly after this, Google will dump this location from its index. It may hold onto it for a bit longer if third-party sites continue to point at it, but not for long.

2. Users referred into the site by external links, to locations that are then superseded by the new organisation of the site, are also presented with a 404 and their journey comes to an end. Some sites present a Custom 404 that attempts to be sensitive to the users' needs, but this is never what the visitor expects and it can often be very hard for them to find the content they were looking for afterwares - if they can be bothered to look - usually they'll head back to Google and look for an easier source.

3. Behind the scenes Google is devaluing the inbound links to the site, as they no longer point to content, but to 404 dead locations. These links will still have some value in Google's eyes, because they point to the domain, but not the value that they once had. The site's performance in Google will start to fall off rapidly as the extent of dead locations is found by Google's crawling activities.

4. Slowly Google is also discovering new locations on the site because of the new technology, but depending on the scale and popularity of the site, it make take a long time to re-index all of it. When it has re-indexed it all it won't then be performing as well, because the site has dumped the value of all of the links into it (other than to the home page) by not respecting their individual locations.

The sad thing is that this scenario is happening daily on big and small sites alike.

AND THE WAY TO AVOID THIS HAPPENING IS ASTONISHINGLY SIMPLE!!!

For every location of the old site create a map to identify where that content is going to be located on the new site. For example:

www.oldsite.co.uk/cms-system-complex-url/article.php?=12345
Will be found at
www.oldsite.co.uk/new-cms/articles/12345

A spread sheet of such locations, which can be converted into rules on bigger sites, can then be given to the implementation team with the following clear instruction:

A. When the server receives a request for one of the old locations, serve a 301 status code (this tells the visitor that the location has moved permanently - it is called a 301 Redirect) and then serve the new location (as defined by the map).

A visiting web user, may not even notice what happens; they'll have clicked on a link on a site or in Google and will have ended up on the right piece of content. The fact that it is now at a different location in the address bar is likely to be lost on them.

A visiting search engine will definitely notice what happens; they will have received a clear instruction that the old location they had for the content is now obsolete, but they should replace it with a new location, as served.

The consequences of this simple change to the outcomes described above couldn't contrast more clearly:

1. Google attempts to re-index the old location and is told that the content is no longer there, but is to be found at a new location permanently. Google will then simply swap out the location information in its index entry for the page and carry on as if nothing had happened.

2. Users seamlessly arrive on the new location for the content and carry on as if nothing had happened.

3. Behind the scenes, Google is conferring all existing reputation from inbound links to the new locations.

4. Google will slowly swap out all the old locations for the new ones as it crawls the site with no noticeable impact on traffic or relevance.

Time and again, such a strategy sees new technology implemented without a moment's blip in the traffic from Google.

Job done.

Testing NOODP. Google is returning the DMOZ title for my site (screenshot) when a search reflects the precise words in my DMOZ listing. So I have added the NOODP meta tag, as below, so that I get the full title in a search for me. Let's see how long it will take to change.

No More DMOZ Titles and Snippets in your Google Listings, hurrah! Today Google announced a new Meta Robots attribute that specifically requests that data from the Open Directory Project is NOT pulled into the Google listing for the site.

To prevent all search engines that support the meta tag from using ODP information (Google's words, not mine), use the following:

<META NAME="ROBOTS" CONTENT="NOODP">

This is a welcome addition, as Google's methodology for the inclusion of this data has appeared inconsistent. Of course, Google's appetite for your pages will determine how quickly they will change, if you are suffering from this condition. If you are not, do it anyway. That's all you need to know for now, so off you go, get on with it!

Small, US-based, Web-Standards-Compliant, Design and Development Agencies Invitation: Now don't all rush at once, but I occasionally need on-the-ground support for my US clients, in the form of local web site design and development services. I have clients currently on both US coasts, so agencies in both the East and West would make sense.

I am only interested in small agency work, usually self-build, with some sensible CMS possibilities, but critically you must be Web Standards compliant and be really up on the importance of Google Optimization. I am not interested in the enterprise space as I do not intend, nor do I imagine, getting involved in new or re-build projects that would require such support. My clients are often deployed on enterprise technology, but I end up working with the encumbent supplier and am not invited to introduce new suppliers. Typical projects will be between $2000 and $10,000, so will suit a small team or a couple of independent practitioners.

If you are interested send me some credentials, some sample sites, and at least two client references. Please don't waste my time if your own site doesn't practice what I am preaching. Thank you for your attention.

Reminder of the importance of Google Images. According to the latest statistics from Hitwise, Google Image Search receives around 5% of all downstream visits from Google UK (it is the same for the .com site too). Let me say that again; 5% of all visitors to Google UK look next at Google Images! Given that Google is the biggest UK web site, 5% is a lot of people.

A quick tip about JavaScript tagging for web analytics: All of these systems require the page carrying the tag to poll the server of the firm offering the service, in order to process the JavaScript. If for any reason these servers are under-performing when the request is made the page load can be delayed, and this is sometimes cited as a reason to avoid such web analytics methods. In my pages you will find two tags, one for Google Analytics and one for IndexTools. The simple solution to this problem is to locate the JavaScript at the bottom of each page and not in the Head, which will allow the whole page to load before the JavaScript is processed.

Latest Hitwise data on the UK search market: Hitwise.co.uk today released one of their regular PR reports on the state of the UK search marketplace. The key findings from a Google optimisation point of view are related to Google's continuing growth in the UK market for web searches. Here are a few of the key statistics demonstrating Google's market share growth:

• Google UK is the most visited website in the UK, receiving more than twice the share of visits compared with MSN Hotmail, the number 2 ranked site in all categories.
• Google UK, MSN.co.uk Search, Yahoo! UK & Ireland Search and Ask.com UK Search together powered 82% of all UK internet searches in the four weeks ending 20th May 2006.
• Combining the UK and com properties for these search engines that number climbs to 96%.
• Google UK continues to dominate the search engine market in the UK, powering more than three-quarters, some 77%, of UK internet searches (combined UK and com properties).
• Hitwise data indicate that visitors to Google are performing multiple searches and using the search engine as a point from which to navigate the web.
• Comparing share of visits with share of executed searches, it is evident that Google powers a much larger share of searches relative to the share of visits the site receives. It is worth noting that within a single session, a user can perform a search, visit results pages, and return to Google to perform another search, so long as the user returns within 30 minutes of the initial visit.
• Visitors to Google UK spent an average of 13 minutes and 30 seconds on the site in the week ending 20th May 2006; this is 2 minutes and 46 seconds longer than visitors spent on Yahoo! UK and Ireland Search, the search engine among the leading four with the second longest session duration.
• As with all search engines, the highest volume search terms on Google UK are navigational. In the four weeks ending 20th May 2006, the highest volume searches on Google UK were "ebay", "hotmail", "bebo", "yahoo" and "argos".
• Whilst Google dominates, the battle for second place is heated with MSN and Yahoo!'s search properties neck-and-neck in share of executed searches. MSN Search powered 7.37% of UK searches and Yahoo! Search powered 7.41% of searches (combined UK and .com) in the four-week period ending 20th May 2006.
• In the US, Yahoo! Search is a stronger competitor. Amongst US internet users, Google.com powered 59% of all searches, compared to Yahoo! Search at 22% in the four-week period ending 20th May 2006. MSN Search powered 12% and Ask.com, 4% in that same four-week period.

Google's continued dominance of the UK search market - in fact it is fast becoming a web location that users start at, even if they know where they are going, because it is easier to ask Google to find the right web address than it is to type it into the address bar - makes your optimisation strategy more and more commercially critical.