Too Few Pages Indexed
If the search engine did not locate as many pages as you expected look here for the most common reasons why.
TOP QUESTIONS
You have not enabled PDF indexing
If your PDF files are not being indexed be sure that PDF indexing is enabled.
To do this simply log in, go to the
page and then use the PDF indexing link.
You are not using the actual address of your site
It is fairly common for sites to use a redirector to "give the site a more memorable name".
If your site is like this, or if it uses a redirector for some other reason, then your account
needs to specify the actual location of your site and not the location of the redirector.
For most sites, you can determine the actual location of your site by waiting
for your home page to finish displaying, then using the address as shown at
the top of your browser's window.
Some "name" services don't redirect but actually place your entire site within a frame
so that the original name is still displayed in the browser's address field. For these
types of arrangements you still need to use the actual location of your site and not the location
of the "name".
Note that you do not need to sign up again to change your site's address!
Simply log in, go to the
page and then use the change URL link.
Your hosting company changed the actual location of your site
This happens more frequently than you might guess (and they usually do it without telling
you!), so double-check that your web site address is still the same.
For most sites, you can determine the actual location of your site by waiting
for your home page to finish displaying, then using the address as shown at
the top of your browser's window.
Note that you do not need to sign up again to change your site's address!
Simply log in, go to the
page and then use the change URL link.
Your site has an advertising frameset
It is common for hosting companies which provide free web space
to place your site within an "advertising frameset". Sites like
these have only one page at your home address - the advertising
frameset. All your pages are located someplace else.
To see if this is being done, you can browse to your online home page
and take a look at the HTML source using your browser.
Look for the frameset tag and, if
found, look for a frame tag
indicating the actual location of your site.
For example if your site is at:
https://hometown.example.com/somesite/
when you look at the HTML you might find something like:
<frame src="https://members.example.com/_ht_a/somesite/" name=contentFrame>
This indicates that your pages are actually at:
https://members.example.com/_ht_a/somesite/
Once you determine where your site is actually located you can specify this location as
an additional starting point for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Your are using a nofollow robots meta tag
The search engine spider honors your "robots nofollow" meta tags.
If your site has one or more pages which contain the tag:
<meta name=robots content="nofollow">
then the links on those pages will be ignored, possibly resulting in incomplete indexing
of your site.
There a couple ways of fixing this:
-
You can override a particular robots meta tag by adding the FreeFind version of the tag
after the robots meta tag.
The FreeFind version has the same possible content values, but has the name "FreeFind"
instead of "robots". Use something like:
<meta name=FreeFind content="all">
-
You can make FreeFind ignore all robots meta-tags on all pages by adding the following tag on the first page the
FreeFind spiders reads:
<meta name=FreeFind content="noRobotsTag">
It should be added before any robots meta-tags are encountered.
There are parts of your site which cannot be found by following links
The spider (indexer) locates the pages of your site by following all the links it can
find on your site which lead to other pages in your site.
It does not just scan your server's hard disk!
If there are parts of your site which don't have links to them but need to be indexed,
you must let the spider know about them.
There are two ways of doing this:
-
You can use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag needs to be placed in
a page that is already found, so the spider can find and processes it.
-
You can provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Note that each starting point is treated as if it were a separate site, not a single page.
After letting the spider know where to start looking for the missing areas of your site, you
will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
MORE QUESTIONS
Your site uses applets for links
The spider (indexer) locates the pages of your site by following all the links it can
find on your site which lead to other pages in your site.
It does not just scan your server's hard disk!
Links embedded in applets (or expressed in applet parameters) cannot be followed by our
spider (nor any other one we're aware of). This means that if all your links to a page
are applet links, that page will not be found.
There are two ways around this problem:
-
The recommended way is to use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag should be placed in
the page(s) with the applet links.
-
A less optimal way is to provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Although each starting point is treated as if it were a separate site and not a single page,
if your site uses applet links exclusively you will end up needing to provide a comprehensive
list of all your pages!
After providing "spider-friendly" links, you
will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
Your site has links exclusively in flash content
The spider (indexer) locates the pages of your site by following all the links it can
find on your site which lead to other pages in your site.
It does not just scan your server's hard disk!
Links embedded in flash content cannot be followed by our spider.
This means that if all your links to a page are flash links, that page will not be found.
There are two ways around this problem:
-
The recommended way is to use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag should be placed in
the page(s) with the flash links.
-
A less optimal way is to provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Although each starting point is treated as if it were a separate site and not a single page,
if your site uses flash links exclusively you will end up needing to provide a comprehensive
list of all your pages!
After providing "spider-friendly" links, you will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
The only links to some pages are built using javascript
The spider (indexer) locates the pages of your site by following all the links it can
find on your site which lead to other pages in your site.
It does not just scan your server's hard disk!
Although FreeFind is unusual in that it tries to extract links from javascript,
if your links are built up out of separate pieces they will not be found.
There are two ways around this problem:
-
The recommended way is to use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag should be placed in
the page(s) with the built-up javascript links.
-
A less optimal way is to provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Note that each starting point is treated as if it were a separate site, not a single page.
After providing "spider-friendly" links, you will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
The only links to some pages are in included JS files
The spider (indexer) never reads JS files, so links your included javascript will not be found.
There are two ways around this problem:
-
The recommended way is to use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag should be placed in
the page(s) with the built-up javascript links.
-
A less optimal way is to provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Note that each starting point is treated as if it were a separate site, not a single page.
After providing "spider-friendly" links, you will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
You have links embedded in some type of browser plugin
The spider (indexer) locates the pages of your site by following all the links it can
find on your site which lead to other pages in your site.
It does not just scan your server's hard disk!
Links embedded in plugin content cannot be followed by our spider.
This means that if all your links to a page are plugin links, that page will not be found.
There are two ways around this problem:
-
The recommended way is to use the
FreeFind Links tag to provide the
spider with links to follow but which are not displayed. This tag should be placed in
the page(s) with the plugin links.
-
A less optimal way is to provide one or more additional starting points for the spider.
To do this simply log in, go to the
page and then use the set starting points link.
Although each starting point is treated as if it were a separate site and not a single page,
if your site uses plugin links exclusively you will end up needing to provide a comprehensive
list of all your pages!
After providing "spider-friendly" links, you will need to re-spider your site to see the results.
To do this simply log in, go to the
page and use the index now link.
Your site is not fully contained in the starting directory
By default, the spider constrains itself to the directory tree that it starts in.
For example, if it starts at your home page:
https://members.example.com/~imauser/index.html
the it will follow links to here:
https://members.example.com/~imauser/archive/...
but not to here:
https://members.example.com/~somebodyelse/...
Typically this is exactly what's desired. In some cases, however, you may want to the spider
to jump to some other directory tree.
For example, if your home page is actually down a directory:
https://members.example.com/~imauser/home/index.html
and you want to follow links to here:
https://members.example.com/~imauser/archive/...
then you will need to let the spider
know that it's OK to jump to the archive directory.
You do this by providing an additional starting point for the spider that is within the
archive directory, i.e.:
https://members.example.com/~imauser/archive/index.html
To set additional starting points simply log in, go to the
page and then use the set starting points link.
Note that each starting point is treated as if it were a separate site and not a single page.
After specifying additional starting points, you will need to re-spider your site to see the results.
To do this use the index now link.
Your site is not fully contained within the starting domain
By default, the spider constrains itself to the domain that it starts in.
For example, if it starts at your home page:
https://example.com/index.html
the it will follow links to here:
https://example.com/archive/...
but not to here:
https://example.net/...
Typically this is exactly what's desired (if it didn't do this, it would end up indexing the entire web!).
In some cases, however, you may want to the spider to jump to some other domain.
An example might be a site:
https://example.com/index.html
which has a separate domain for the archive area:
https://archive.example.com/index.html
In cases like these you will need to let the spider
know that it's OK to jump to the other domain.
You do this by providing an additional starting point for the spider, like:
https://archive.example.com/index.html
To set additional starting points simply log in, go to the
page and then use the set starting points link.
Note that each starting point is treated as if it were a separate site and not a single page.
After specifying additional starting points, you will need to re-spider your site to see the results.
To do this use the index now link.
There was internet congestion when you last indexed your site
Yes, it happens. Sometimes the internet has a traffic jam!
Although the search engine spider (indexer) will try repeatedly, sometime it just
cannot get through to your server. The solution is simple: just respider your site.
Before you do it is often a good idea to make sure that your site is actually online.
Go online to your home page and "refresh" the page to make sure that everything seems
to be working.
Your site is up? Then log in, go to the
page and then use the index now link to respider your site.
|