Fast File Searching With Indexing Service

Indexing Service is part of Windows 2000 and Windows XP. Once activated, it works in the background to maintain an up-to-date index of all the files in selected areas of the computer’s hard disk. What makes it especially useful is that it keeps details of the content of the files as well as more mundane information like filename, date, size, etc. As a result even when you search for a file by content the results come up almost instantly whereas without it the search can take minutes and is usually not worth bothering with.

Considering its power and usefulness, Indexing Service is a surprisingly neglected feature. This may be because:

- The search interfaces provided in Windows are clumsy and don’t do it justice.

The first two issues are easy to solve and a nice new third-party interface, AimAtFile Fast File Search, takes care of the third item. I hope these notes will help with the fourth. All the examples have been tested on my own computer. The fact is that you can get results from Indexing Service very easily and with hardly any performance penalty. Indexing Service has transformed the speed and ease with which I can access my own information.

Search Interfaces

Windows comes with two interfaces to Indexing Service already built in but a much better one is now available as third-party software. My views on the three options are as follows, but you can easily decide for yourself by downloading AimAtFile Fast File Search for free trial from aimingtech.com.

- Indexing Service Query Form is fairly rudimentary and it seems to be intended as an experimental tool rather than a practical interface. Access is hidden away in Manage / Services and Applications / Indexing Service (right click on My Computer to see Manage) and you may need administrator privileges to open it. There is only one entry box, so pinning a query down to a particular directory or file type involves multiple search terms plus a bit of a learning curve. Also the format of the results makes this interface unattractive for normal use. Nevertheless it works well enough to give a tantalising taste of what Indexing Service can do.

- Windows Search is easy to access, e.g., from Start / Search / Files and Folders, and you can activate Indexing Service from its main window. Afterwards, and once Indexing Service has had time to index the data, search results from Indexing Service appear very quickly, followed considerably later by a second set of results produced the hard way by trawling through the hard disk. Perhaps some users will be happy with it, but I found it confusing and restrictive. Sometimes it gives unexpected results and some directories that I assigned for indexing remained invisible to it even though they were visible to Query Form or AimAtFile Fast File Search. The important option of presenting the output in order of rank (see later) does not work and all the settings go back to default every time you reopen it. The article at xpsearch describes how to use an undocumented feature to force the search to restrict itself to Indexing Service. This works ok for some search expressions but not for more complex ones as brackets are not supported.

- AimAtFile Fast File Search nicely addresses most of the limitations of the built-in interfaces. As well as good presentation of the output listing, it provides three separate boxes to enter the search words (Query), the type of file (File), and the root directory to search in (Folder). You can get useful results with hardly any learning effort – just type a few words into the Query box, and perhaps a file extension into the Files box, and click Search. One addition I would love to see in future revisions is the ability to select a group of files and delete or drag them elsewhere. Another is the ability to view the abstracts that Indexing Service can generate. However the current version is the first so there’s plenty of time for enhancement.

I assume the use of AimAtFile in most of what follows. However by using combined queries (see later) you can do all the same things with Query form instead, but at the price of more complex search expressions and poor presentation of the output listing.

Tune for Best Results

You don’t have to do it, but if you address the three items below Indexing Service will require less computing resources than if you stick with the default settings. All this is easy to do and details are available at xefteri.com and pcworld.com

- Use the tune-up option in Indexing Service and set the ‘Indexing’ slider to maximum. This way the index is updated immediately a file is saved or deleted. Otherwise it gets scheduled for later when the computer is idle.

- By default Indexing Service monitors nearly all the files on the computer. This wastes resources. It makes sense to restrict monitoring to the folders that you want to search in, such as My Documents.

- Leave the option to ‘generate abstracts’ unselected (Indexing Service on Local Machine / Properties). There’s no need to waste resources on abstracts because there seems to be no way to access them with the available interfaces. But if AimAtFile Fast File Search ever gets a ‘preview pane’ that can display the abstract of the selected file, this is where you would come to turn on abstract generation.

- If your file system uses FAT or FAT32 consider changing to NTFS. Indexing Service works best with NTFS because there’s less work for it to do and the catalogues use less disk space.

Basic Searches Are Easy

Free-Text Search

The simplest searches involve ‘free-text’ and ‘phrase’ queries. In the first you enter one or more words into the Query box. In the second you put them in quotation marks. Either way, if you want to further qualify the search and restrict it to files from say Microsoft Word, then you can enter the file extension, ‘doc’ in this case, into the Files box.

When you type a string of words without quotation marks into the Query box and click Search, the following things happen.

- Indexing Service first discards any words that are on a list of ‘noise words’. These are listed in Help and include such relatively uninformative words as ‘about’, ‘for’, ‘them’, ‘how’ as well as single letters and single numerals. The words ‘or’, ‘and’, ‘and not’, and ‘near’ are special. If they appear between other words they behave as Boolean operators (see box), if not they are treated as noise words.

- The search then looks for files that contain as many as possible of the remaining words together with their extended or inflected versions. An example of extensions is that if you enter ‘dog’ it will also look for ‘dogs’ and if you enter ‘dogs’ it will also find ‘dog’. An example of inflections is if you type in say ‘flew’, you will get hits from files containing any of ‘fly’, flew’, ‘flies’, or ‘flown’. It doesn’t matter which of the possible inflections you enter, the full set are used in the search.

- Each file in the output is assigned a numerical ‘rank’ proportional to how well it matches the query. This is a useful feature (especially when the operator ‘near’ is used) because if you set AimAtFile Fast File Search to order its output in ‘rank descending’ you get to see the most relevant results first. [Indexing Service Query Form can also do this. Search Assistant claims to do it but the feature doesn’t work].

Phrase Search

When you enter a phrase (a string of words surrounded by double quotation marks), and click Search, the search goes as follows.

- The search then looks for files containing a text string that includes every entered word in the same order, and that has a word (any word) in each placeholder. Extensions and inflections are ignored – after all, the whole point of a phrase is to specify exactly what you want.

For example, the entry ‘"indexing service"’ will find only files that contain ‘indexing’ immediately before ‘service’. Likewise ‘"computer indexing service"’ will find just those three words in that specific order. If the phrase contains a noise word like the ‘and’ in ‘"computer and service"’, hits will include such as ‘computer indexing service’ and others with any other word substituted for ‘and’. In effect noise words in phrases behave as wildcards standing for ‘any word’.

- To mark a phrase it is important to use “double” quotation marks not ‘single’ ones. The latter are just ignored.

Constructing Short-Form Query Expressions

The simple searches described so far looked for items in the contents of files. However ‘contents’ is only one of many searchable ‘properties’. Five others are listed below but there are many more (see Help).

Property name	Description
All	All properties, including Contents. Works only for text queries, not numeric-value queries.
Contents	Words and phrases in the document
Filename	Name of the document
Size	Document size, in bytes
Write	Date and time the document was last modified

We now look at four basic kinds of query that Indexing Service can apply to these properties: free-text queries, phrase queries, pattern-matching queries, and relational queries. (Vector-space queries are outside the scope of this article but are described in Help).

The general procedure is to write one or more so-called ‘short-form’ queries and combine them together with Boolean operators into a single expression. Each query defines a particular way to search within a particular ‘property’.

A short-form query looks like ‘#filename = *.doc’ and contains the four basic sections shown in the diagram and which are described in more detail below.

Mode

Property

The ‘Contains’ or ‘Equals’ Operator

Input

($, @, or #)

The Property in which to search

(e.g., Contents, Filename)

space (optional)

(‘ ’, or ‘=’)

[or a relational operator e.g. >].

space (optional)

Item(s) to search for, such as a string of words.

[or the target data for a relational operator]

Mode

This symbol specifies whether the input is treated as free-text ($), a phrase (@), or a regular expression (#).

- Free-text is a series of words, and the search finds documents where the specified property contains one or more of the input words (or their extensions and inflections) in any order, but not necessarily all of them.

- A phrase is a series of words in a specific order and the search looks for these exact words in the exact same order. With free-text and phrase queries, any wildcard symbols (see box) in the input are ignored if the ‘contains’ operator is applied but they behave as wildcards if the ‘equals’ operator is used.

- A regular expression is a set of symbols comprising letters, numbers, wildcards, and/or Boolean operators. For the really serious searcher there are also various other operators that will not be considered further here – more details are in IS Help. If the input contains ‘*’, ‘?’, or ‘|’, a short-form query not involving ‘contents’ or ‘all’ gets treated as a regular expression regardless of what mode is indicated.

Property

Of all the properties, ‘contents’ and ‘all’ are special in that they cannot be searched using a regular expression query. This means that an expression containing ‘#contents’ will be rejected as an error. The same happens with ‘$contents =’ and ‘@contents =’ but here the reason is that relational operators (=, <, >, etc.) cannot be used with ‘contents’ or ‘all’.

Contains or Equals

If the ‘equals’ operator appears, to register a hit the value of the property must be the whole word or phrase and nothing but the word or phrase. On the other hand, the ‘contains’ operator requires only that the search word or phrase appears somewhere within the value. For example, ‘@filename= readme*’ would find the file ‘readme.doc’ but not ‘backup of readme.doc’. In contrast, ‘@filename readme.*’ would find both.

- The ‘contains’ operator need not be explicitly written. Its presence is assumed if the space is left blank.

- With properties such as ‘size’ and ‘write’ all the other relational operators besides ‘equals’ can be used. They are listed in the table.

Input

With free-text and phrase queries, the input will be the words or phrases to be searched for. ‘Words’ are groups of letters and/or numbers. When relational operators are used the input will contain a number or an expression. For example ‘#size > 100000’ will find files bigger than 100,000 bytes, and ‘#size>1000000 & <2000000’ will find files between 1 and 2 megabytes.

- If the ‘equals’ operator appears, wildcards can be used. If not, they are ignored (or occasionally do something unexpected). Thus Indexing Service treats ‘@filename *.txt’ as if it were ‘@filename txt.

- If the ‘contains’ operator appears, only words and numbers are recognised and Boolean operators and single letters or numerals are treated as noise words.

Special Cases

To help to minimise any confusion about what can be used with what in short-form expressions, here is a summary of the important special cases mentioned in Help.

- The relational operators (=, <, >, etc.) cannot be used with ‘contents’ or ‘all’.

- ‘Contents’ and ‘all’ cannot be searched using a regular expression query (#).

- If ‘=’ is present, wildcard characters can be used. (This will never happen when the property is ‘contents’ - see first item above).

- With free-text and phrase queries ($ and @), only words and numbers are recognised. Boolean operators, proximity operators, and wildcards are ignored.

- If a short-form query not involving ‘contents’ or ‘all’ contains ‘*’, ‘?’, or ‘|’ it is automatically treated as a regular expression regardless of what mode is indicated.

It seems best to leave wildcards out of free-text queries entirely and leave them out of phrase queries except when ‘=’ is present. This avoids uncertainty about how they will be interpreted by expressions that don’t expect them (as in the last two items in the table).

For anyone who is interested, this table shows how these rules worked out when I tested real examples.

Written As	Behaves As	Reason
$contents=anything	error	‘contents’ (and ‘all’) cannot use relational operators
@contents= anything	error	ditto
#contents anything	error	‘contents’ not usable with #
@filename=*.fax	#filename=*.fax	‘*’, ‘?’, or ‘\|’ turns $ or @ into #
@filename *.fax	@filename fax	wild cards ignored with @ since no ‘=’
@filename fa?	@filename fa	ditto
@filename *doc	@filename doc	ditto
@filename doc	@filename doc	ditto
@filename= fax	@filename fax #filename fax #filename= fax	Presence of ‘*’ makes @ become #
@filename d?m.*	error	Wild cards are ignored with @ since no ‘=’, since single letters count as noise words, this means that no words are left.
The examples below seem to break the rules
@filename com*	#filename com #filename= com (Finds command as well as *.com)	Property not ‘contents’ therefore ‘’ should turn it into ‘#filename fax’ but doesn’t.
$filename= *.fax	$filename .fax @filename .fax	Property not ‘contents’ therefore the ‘’ should turn it into #filename= .fax but it doesn’t. It behaves as if the ‘=’ is removed as well.

Single Queries

As discussed already, typing ‘how do I enter a search query’ into the Query box would define a ‘free-text’ type of search that would attempt to find any or all of the seven words. We can now see that an alternative short-form query for this would be ‘$contents how do I enter a search query’.

Putting inverted commas around the same group of words is equivalent to entering the phrase query ‘@contents how do I enter a search query’. As discussed earlier, and allowing for the noise words, this will retrieve phrases like this: “<any word> <any word> <any word> enter <any word> search query”.

The fact that you can get these results just by typing in the search words themselves means that the full short-form versions of queries using ‘contents’ are seldom required.

Combining Queries

Earlier we discussed the effect of entering ‘computer and disk and not laptop’ into the Query box. We can now see that, written out in full using short-form queries, this would become ‘($contents computer) and ($contents disk) and not ($contents laptop)’. When separate queries are combined like this:

- The Boolean operators between the separate expressions are not optional; they are essential and if missed out will cause an error message. ‘Near’ is not appropriate here so the options are ‘and’, ‘or’, and ‘and not’.

- If there is any possible ambiguity each expression has to be enclosed in brackets. If not, you can leave out one or more sets of brackets as appropriate, but ambiguities are easy to miss and you only need to miss one or two to wipe out any time saved through abbreviating.

- Provided you enclose it in brackets, one or more of the expressions can be a simple word string or phrase such as you might type directly into the Query box on its own.

Here’s an example: ‘(#filename = *printer*.d??) &! (laser)’. The first part is a regular expression query that looks for files where the filename contains the word ‘printer’ with any or no text on either side of the word and where the extension is any three letter word beginning with ‘d’, such as ‘doc’ or ‘dot’. The second part does what typing just ‘laser’ into the entry box would do – it looks for files that contain the word ‘laser’. The combined expression lists files that match the first query, but not if they also match the second. Replacing the contents of the second brackets with ‘$contents laser’ would give the same result.

Here’s an example of a phrase in a combined expression: ‘(#filename = *printer*.d??) & (“colour laser printer”)’. Again the second section acts in the same way as if it were the only item in the entry box. In this case replacing the contents of the second brackets with ‘@contents colour laser printer’ would give the same result.

‘Files’ and ‘Folder’ Boxes In AimAtFile Fast File Search

Using the Query Form in Indexing Service you always have to write dual queries when you want to search in two properties (say content and filename), or triple ones if you want to add a third (say root folder). The fact that AimAtFile has a separate box for each of these three items makes it much easier to use.

As an example, to do the same search as in the previous example, ‘(#filename = *printer*.d??) & (“colour laser printer”)’, you would just type ‘”colour laser printer”’ into Query and ‘*printer*.d??’ into Files. If you wanted to specify a root folder as well, you would either type it in directly or enter it using the browse facility.

- First the contents of the Query box (just as written) and the contents of the Files boxes (processed as below) are combined to get the query expression ‘(Query entry) & (processed Files entry)’.

- If the Folder box is empty, that’s it. If not, the contents of Folder are used as a ‘Scope’ property and combined into the Indexing Service search using the AddScopeToQuery method.

A conceptually simpler way to do this would be to include the Folder specification using a short-form expression such as ‘& (@directory = *\personal\family)’. However the AddScopeToQuery method is more general especially where a LAN is to be searched. Details of AddScopeToQuery are beyond the scope of this article but more information is available in the MSDN Library. Examples 6 and 7 below demonstrate the use of ‘@directory’.

- If Files contains one or more file masks separated with Boolean operators, they are automatically incorporated into one or more regular expressions like ‘#filename= filemask’ separated by the same Boolean operators. Thus the entry ‘*.txt or *.doc‘ becomes ‘(#filename= *.txt) or (#filename= *.doc)’.

- If Files contains short-form expressions, they are used without alteration. (This means that whenever you are writing short-form expressions, singly or in combination, it actually makes no difference whether you type them into Query or Files, or whether you split them between the two boxes).

Note

1. If Query and Files are both empty nothing happens when you click Search. This matches the behaviour of the Indexing Service Query form.

2. If Files contains an entry, but not otherwise, the extra query ‘& (#size>0)’ is automatically added to the combined query. Although this places no practical restrictions on what files will be retrieved it overcomes the bug described below in ‘Undocumented Surprises in Indexing Service’.

Examples of Multiple Queries

The following examples highlight various formats that might be useful. Each one introduces an extra type of expression and/or a modified form of a previous one. Remember to use brackets where necessary to avoid ambiguities.

1. ‘Laptop & computer’. This finds files containing both ‘Laptop’ and ‘computer’.

2. ‘(Laptop near computer) & @write > 2003/2/14 13:00:00’. This finds documents containing ‘laptop’ and ‘computer’ (if less than 50 words apart) and which were modified after February 14th 2003, at 13:00 Coordinated Universal Time

3. ‘(Laptop near computer) & (@write > 2002/1/1 & < 2003/1/1)’. This finds documents containing ‘laptop’ and ‘computer’ (if less than 50 words apart) and which were modified between January 1st 2002 and January 1st 2003.

4. ‘(Laptop and computer) & @doccreatedtm < 00/2/14 & @size <= 100000’. This finds documents containing ‘laptop’ and ‘computer’ and which were created before February 14th 2000, and are less than or equal in size to 100,000 bytes.

5. ‘(Laptop ~ computer) & @write > -1y & (@size < 100000 | > 300000)’ This finds documents containing ‘laptop’ and ‘computer’ (if less than 50 words apart), modified within the past year, and which are less than 100,000 or greater than 300,000 bytes in size. You can also write this out the long way if you want: ‘(laptop ~ computer) & @write > -1y & (@size < 100000 | @size > 300000)’.

6. ‘(Laptop ~ computer) & (@write > -12m & <-6m) & (@size < 100000 | > 300000) & @directory= *pe*’. This finds documents containing both ‘laptop’ and ‘computer’ (if less than 50 words apart), modified between twelve and six months ago, and which are less than 100 or greater than 300 kilobytes in size, and where the physical path contains a directory with ‘pe’ as part of the name (e.g., ‘personal’ or ‘spec’).

7. ‘(Laptop | computer) & @write > -300d & (@size > 1000000 & < 1800000) & @directory= *\personal\fam*’. This finds documents containing either ‘laptop’ or ‘computer’, modified within the past 300 days, between 1 and 1.8 megabytes in size, and which are in any subdirectory of \personal\ that begins with ‘fam’ (e.g., family). [NB. You must use the backslash with @directory – a forward slash won’t work].

Searching for Filenames

The following examples work fine in the Query box of either Query form or AimAtFile Fast File Search. However with the latter it’s easier to put the filename mask(s) directly into the Files box leaving the rest to go into the Query box.

These examples focus on the filename part of the search. Remember that any other expressions can also be added as well provided they are suitably bracketed and separated by ‘and’, ‘or’, or ‘and not’ as appropriate.

Examples of Filename Searches

1. ‘#filename= *.doc’. This will retrieve all files that have ‘doc’ as the extension.

2. ‘(#filename= *.doc) & (computer)’. This finds all doc files that contain the word ‘computer’. [This expression requires at least one set of brackets].

3. ‘(#filename= *.d??) & ("all rights reserved")’. This finds documents with extensions like dot or doc that also contain the phrase “all rights reserved. [This expression requires at least one set of brackets].

5. ‘(#filename *.|(d??|,t?t|)) & ("all rights reserved")’. This is a concise (but cryptic) way of writing the previous example using the more advanced style of regular expression mentioned in ‘Mode’ earlier. Refer to Help for details.

6. ‘(#filename= *.doc) & (computer) & (laptop) or (#filename= *.txt) & (computer ~ tandon)’. This finds doc files containing both the words ‘computer’ and ‘laptop’ anywhere in the file. It also finds txt files containing ‘computer’ and ‘Tandon’, but only if they are less than 50 words apart.

7. ‘#filename= doc*’. This finds any file where ‘doc’ appears as the first letters in the complete filename, e.g., ‘document collection.txt’.

8. ‘@filename doc’. This finds any file where ‘doc’ appears as a discrete word anywhere in the filename or extension, e.g., ‘list of doc files.wbk’.

Undocumented Surprises in Indexing Service

Indexing Service occasionally plays ‘jokes’ like these. The ‘Hint’ below explains how to overcome them.

- If ‘#filename= *print*.txt’ is the only entry, it works fine but change it to ‘#filename= *printer*.tx?’ and no files are found at all (which can be a shock if you know they are there). Adding another query to search in a second property as well will usually make it work as expected – see Hint.

- When there are no other entries ‘#filename= *print*.txt’ and ‘#filename= *print*.doc’ work fine when used alone, but if you combine them into the expression ‘(#filename= *.txt) or (#filename= *.doc)’, or its equivalent ‘#filename *.|(doc|,txt|)’, no files are found at all. Adding a second entry works here as well – see Hint.

- In contrast, ‘@filename *.txt’ and ‘@filename *.doc’ work fine either alone, or when used together in the expression ‘(@filename *.txt) or (@filename *.doc)’.

Hint : If you are using Query form and you get the false result ‘no files found’, try adding ‘& #size>0’ to the combined search term. This seems to bypass the above problems yet adds no effective restriction to the search. (AimAtFile Fast File Search applies this trick automatically).

Where to Find More Information

The aimingtech.com website is an excellent useful source of information about Indexing Service and includes details of how to tune it for best results. So does the article at xefteri.com where the author rightly comments, ‘documentation on <indexing service> is amazingly scarce and scattered.’ Although aimed mainly at users setting up Indexing Service for indexing a website, the detailed screenshots and general information apply equally well to anyone wishing to tune it for their own purposes.

An article at PC World contains a useful guide to how to start Indexing Service and to make it focus on the files you need. More useful information on this aspect appears at xefteri.com.

Although the treatment is incomplete, the article at xpsearch contains useful information about how Indexing Service relates to XP Search Assistant. The author describes what seems to be an otherwise undocumented trick of forcing Search Assistant to restrict itself to Indexing Service by putting an exclamation mark before an entry in the Containing Text window.

Most of the published material on how to actually use Indexing Service to do searches seems to be intended for developers rather than users. Most of my information came from the Help files in Windows XP together with lots of experimentation using a small set of carefully selected files.

You can easily access Help for Indexing Service by clicking on the logo at the top left of the AimAtFile search window and selecting Indexing Service Management. Otherwise you need to right click on My Computer and select Management, etc.

Amplification of certain points came from the Microsoft MSDN Library. A search for ‘Indexing Service’ or ‘Computer Indexing Service’ at the same site will bring up further specialist information.

David Tong, Leeds, UK.
After ten years of academic research in chemistry, the author spent the next two decades running his own specialist electronics company. Nowadays when not playing with computers he plays the accordion. He has no formal connection with AimingTech.