Comments on: Is it time to abandon Google search for docs?
REBOL Technologies

Comments on: Is it time to abandon Google search for docs?

Carl Sassenrath, CTO
REBOL Technologies
26-Apr-2010 22:43 GMT

Article #0471
Main page || Index || Prior Article [0470] || Next Article [0472] || 5 Comments || Send feedback

Over the last year during the major rework of both the REBOL website and documentation (hundreds of pages) we selected Google as the search engine for the site. However, Google's been doing a poor job for some unknown reason.

Let me give you an example:

Today, while working on R3 Docs, I searched for the construct function page. Google displayed four results. None of them were the Construct Function page (which has been there for more than a year, so Google's had plenty of time to index it.) Similarly, when I search for "compose" I don't see the compose function listed.

This is a problem. Many of us depend on the web site's search feature to find the pages we need. Note, I'm not trying to bad-mouth Google here, I just need it to work really well for our web pages.

Those of you with admin access to the R3 Docs know that you can use a REBOL-based search function from the admin page. It uses the same basic search code as this blog, which is by no means fancy, but it tends to give good results. Searching for "construct", I get:

  1. Functions: construct  
  2. Datatypes: Char!  
  3. Functions: make  
  4. Guide: Basic syntax  

The top result is what I was seeking.

So, we've got ourselves a bit of a conundrum. To figure out what's wrong with Google (or perhaps the way we're presenting the pages to it), or drop it as the primary search engine and find an alternative (how's RIX doing these days?)... or use our own code.


PS: Adding this blog with the above link may cause construct to become better indexed in Google, but you can still find various R3 functions that are mysteriously missing from Google search.


Tried a few other engines to search for construct, here's a summary of results (in alpha order):

  • Alltheweb - found the R3 page and a few related pages.
  • AltaVista - found the R3 page and a few related pages, same as above.
  • Bing - found the R3 page and a few related pages, same as above (surprised!)
  • Cuil - found the R3 page (top result) but very few related pages (derails totally by third result.)
  • Gigiblast - horrible.
  • Google - failed to find the R3 page
  • Lycos - failed to find the R3 page
  • RIX (REBOL specific) - failed to find the R3 page
  • Yahoo - failed to find the R3 page

Because this website is easy to modify, we'll test out the Bing search engine for a while. Let's see if it does better.



27-Apr-2010 1:44:51
First of all you should provide a sitemap. It's the best way to let the search engines know what is important and what is new (without need to crawl all the pages again and again).

Here is some how to:

It's easy, it's just a simple XML file. Here is some more reading:

27-Apr-2010 1:53:14
And if I can have one more note: The I would choose reverse order in the page TITLE. For example instead of:

REBOL 3 Functions: construct

I would choose something like:

Construct / Functions / REBOL 3

The reason is, that if you have a lot of tabs opened in the browser, you see just the beginning of the title. If I have opened more REBOL related tabs, I see just REBOL many times.

Just go and check Wikipedia - notice, that they don't put word Wikipedia at the beginning of the title as well.

27-Apr-2010 2:21:44
Also we have a very nice service related to SEO optimization. Check this:

It's only in czech language, but you can see what's the main problem: there is no description, no keywords, no sitemap and there is no robots meta tag in the header, for example:

27-Apr-2010 2:35:11
The empty code above is:
<meta name="robots" content="index, follow">
Carl Sassenrath
27-Apr-2010 19:04:19
Hi Oldes, yes, there's always been a sitemap for those doc pages, see -- and Google knows about it... but, like so many other myths-of-search, it really makes "no difference". Proof: Bing finds all the pages just fine, and it doesn't even know the sitemap's url.

Also, it's another myth that modern search engines use the HTML metatags. They haven't for many years. Reason: those tags are not "trustworthy" and the SE's find the actual content of pages to be more accurate. However, Google will use the description line for summary text in its results, but we only do that on the primary page.

Regarding title... perhaps we should, but then it's also reversed in the search engine summary page, which isn't a good thing. So, that's a bit of a trade off.

Post a Comment:

You can post a comment here. Keep it on-topic.


Blog id:



 Note: HTML tags allowed for: b i u li ol ul font p br pre tt blockquote

This is a technical blog related to the above topic. We reserve the right to remove comments that are off-topic, irrelevant links, advertisements, spams, personal attacks, politics, religion, etc.

Updated 8-Jun-2023   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code