Searching Forums

Forums are a popular part of many websites and, with a little care, can be included in your site's index without problem.

Contents Overview Determining the "Good" and the "Bad" Excluding Pages Ultimate Bulletin Board UltraBoard vBulletin Ikonboard Discus Board Power MSN Messageboard This tutorial is not a web/html primer and assumes that you already know how the process of "web surfing" is accomplished (i.e. a browser requests a page from a server which then returns the page to be viewed), what an HTML "form" is and how it works, and what a link "target" is. If you are not familiar with these concepts please read a basic web/html primer.

Overview

Like many types of dynamically-generated site content, forums include many types of pages that you do not want to include in your index. Examples might include topic reply, registration, edit post, new topic, delete post, and send thread pages. Often these "junk" pages outnumber the good pages 10-to-1! If you do include these pages you may end up indexing thousands of extra pages and your search results may contain irrelevant results.

In order to precisely index your forum you will need to:

Determine the "good" parts of your forum (i.e. the parts you want indexed),
Determine the "bad" parts of your forum (i.e. the parts you do not want indexed), and
Exclude the search engine spider (indexer) from the "bad" parts of your forum.

The following sections cover these steps.

Determining the "Good" and the "Bad"

The way we usually do this is by simply going to the forum and looking! Using a text editor, I create a file with "good" and "bad" section headings in it. Then I go to the forum and gather an example address for each type of page the forum has, placing each address in the proper section (good or bad). Typically you end up with a couple good addresses and lots of bad ones.

Another good way of finding some (but not all) bad pages is to search your site for "password". Many administration type pages require the user to enter a password, so this search can show these types of pages.

When you have a your list together, then you can move to the next section: Excluding Pages.

Excluding Pages

For detailed help on how to exclude pages from your index see Excluding Pages using the Control Center.

Here is a quick example for review. If you had these exclusions:

		/test/*
		/cgi-bin/postings.cgi?action=reply*
		/a*

then the following addresses would be ignored by the spider:

		http://example.com/test/index.html
		http://example.com/cgi-bin/postings.cgi?action=reply&id=123
		http://example.com/cgi-bin/postings.cgi?action=replytothis
		http://example.com/abc.html

and the following ones would be allowed:

		http://example.com/test.html
		http://example.com/cgi-bin/postings.cgi?action=edit
		http://example.com/cgi-bin/postings.cgi
		http://example.com/bbc.html

It is also possible to first disallow a whole section of your site (the forum) then selectively "allow" certain parts to be included in the index. For example:

		/cgi-bin/* index=no follow=yes
		/cgi-bin/Ultimate.cgi* index=yes follow=yes
		/cgi-bin/forumdisplay.cgi* index=yes follow=yes

These exclusions prevent the spider from accessing every cgi-bin address from being accessed except Ultimate.cgi and forumdisplay.cgi.

Using exclusions structured in this manner can often simplify things.

Now that you know how to prevent pages from being indexed, and you have a list of "good" and "bad" page types, you will need to create your list of exclusions. You do not use the entire sample addresses, but just enough to uniquely exclude the type of page.

The following sections have exclusion lists for some popular forums.

Ultimate Bulletin Board

If you are using version 6 or above, use the following exclusions as a starting point for indexing this forum:

		/cgi-bin/ultimatebb.cgi*
		/cgi-bin/ultimatebb.cgi index=yes follow=yes
		/cgi-bin/ultimatebb.cgi?ubb=forum* index=yes follow=yes
		/cgi-bin/ultimatebb.cgi?ubb=get_topic* index=yes follow=yes

If you are using version 5 or below, use the following exclusions as a starting point for indexing this forum:

		/cgi-bin/ubbmisc.cgi*
		/cgi-bin/Ultimate.cgi?*
		/cgi-bin/postings.cgi*
		/cgi-bin/archive.cgi*
		/cgi-bin/search.cgi*

You may have to change these examples to match your exact configuration.

UltraBoard