Synapse Hotline X Daemon Devin Teske Request for Comments: 3 Revision 1 Category: Informational Re: shxd July, 2007 Improved File Searching Status of this Memo This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Abstract This memo specifies guidelines for implementors and users of the hotline protocol and how to implement a better file searching routine that is backward-compatible with all hotline clients (1.2.3 compatible or higher) and forward-compatible with all new/enhanced Hotline clients. Ackowledgements This memo is the result of thinking straight. 1. Introduction File searching is not built into the Hotline Protocol. The ability to search for files has been implemented in various ways. Some of the ways that file searching has been implemented under Hotline are (including but not limited to) bots, chat commands, and custom software solutions. Each of these implementations is rarely intuitive and even more rare, useful and/or efficient. In the following section I will explain the problems with each implementation and outline in brief a better way to implement file- searching in Hotline. 2. Summary of Recommendations Each of the currently implemented solutions for file-searching in Hotline has its deficiencies. I have listed below, three common implementations, a description of how they work and the problem(s) with each implementation. Following the analysis of current implemen- tations, I will explain how I intend to solve all of the below mentioned problems. Implementation: Bots How it works: A user private-messages a bot in the userlist with a specific syntax to indicate that he/she would like to search for a file. After the bot searches for the file (either in a pre-computed database or live-search), it responds to the user with a list containing the path to each matching file. Problem(s) with implementation: - The user has to navigate to each file by pathname to be able to operate on that file (download, delete, etc.). - The search results are not saved, so the user must keep the response from the bot (beit private-message or chat, etc.) or the paths to matching files is lost. - Very little (if any) information about each matching file is given by the response from the bot (for example, file size). - Requires that a bot be logged into the server. Bots can be unreliable because they require an additional piece of software to run and (if logging in from anywhere besides localhost) a separate network connection. - Generally, you are limited to running a bot that is compatible with the operating system that the server runs on. Many bots are written for Windows only, so if you run a UNIX server, you either have to run a Windows box on another network or try and find a bot that is compatible with your server OS. - Many users would rather point-and-click rather than interpret a list of pathnames. Implementation: Chat-Commands How it works: A user sends a chat line similar to "/find foo" and the server searches for "foo". The results from the file search are [usually] a list of files paths returned to your chat window. Problem(s) with implementation: - Chat from other users crowds the chat window and can be mixed with the results from the file search. - The user has to navigate to each file by pathname to be able to operate on that file (download, delete, etc.). - Very little (if any) information about each matching file is given by the response from the server (for example, file size). - Chat from other users is constantly shifting your search results out of view. - Many users would rather point-and-click rather than interpret a list of pathnames. Implementation: Custom Software Solution How it works: A skilled programmer modifies an existing client/server (or writes his/her own from the ground up) to add a file searching routine. Problem(s) with implementation: - Requires a user to run the custom client to benefit from the added feature. - Not backwards-compatible with the myriad of other clients available. How an enhanced file search might be implemented: Properly implemented, enhanced searching will allow a user to perform a search, then browse the list of results using the standard file browser built into Hotline Client Connect suite (or any other client for that matter). By populating a fake directory in the file list with the results of the most recent file search, a user can directly operate on the files and folders matching their search. A standard search procedure might look something like this: 1. Enter a chat-command in the form of "/search foo" in either a public or private chat window. 2. You will be informed by private-message when the search has been completed. The private-message will include instructions to the user to navigate to "/Search-Results" within the file heirarchy. 3. The user will open a file list and navigate to the top-level- directory named "Search-Results". 4. In that directory will be a list of files and folders that were found to match the user's search criteria. 5. The client software will be unaware that the files in this directory are actually links to the originals. Thus, any action that can be performed on the original may be taken on the link. 6. When the user disconnects or performs another search, the previous results may be cleared. This implementation is much more intuitive, efficient, and useful than the aforementioned methods. Naturally, when a user searches for a file/folder, it should be assumed that they intend to act upon that item once it is found. Therefore, it only makes sense that we show the results in a file list so that the user can act upon it without any additional steps (such as having to navigate to the item first). This implementation (over the current ones) has the added benefit that it can still be used by any client that accurately implements chat and file-handling. 3. Implementing enhanced file searching There may be multiple ways that a programmer can add this functionality to pre-existing server software. I will outline two different methods in this section. In the next section, I will explain some of the implementation caveats and how to work around them. The first way that you can implement this feature is to rewrite the server software. To do this, you will have to accomplish the following programming tasks (in brief): 1. Rewrite the chat-handling routine to interpret the "/search" command for initiating a file search. 2. Rewrite the file-list routine to dynamically add a non- existant directory named "Search Results" to the top-level- directory listing. 3. Write the file-search routine. This routine should take (at least) a filename to search for. After searching for a match, the results should be stored for later retrieval by the file- list routine (see task number 4). 4. Rewrite the file-list routine to list the results from the most recent file-search when requesting a listing of "Search Results" directory (see task number 2). The second way that you can implement this feature is specific to the hxd server software. Under hxd (and hxd derivatives), you can enable a feature called "exec", which allows you to execute a shell script with a given name by typing "/NAME" in chat (where "NAME" is the name of the script). By using the exec feature of hxd, you can add the enhanced file-searching feature with fewer changes to the server software. The following programming tasks will need to be completed to achieve this (in brief): 1. Write a shell script to implement the file-searching routine. The script should take (at least) a filename to search for. 2. Maintain a temporary directory where search results can be held. The latter implementation still requires a minor modification to the server software. I will discuss the required modifications in detail in the following section. Each method of implementation has benefits. They are listed below. Implementation: Modification of Server-Software Code Benefits: - Faster implementation (depending on the language that the server software was made in; for example, C is faster than Shell scripts which are used in the other implementation). - Better event handling. We can perform more event-driven tasks such as clearing search-results for a particular user when they disconnect. The other implementation method does not have access to the server internals (such as disconnect events, etc.). - Can be implemented on more platforms. The other method we are going to discuss makes use of filesystem links. Not all operating systems and filesystems support symbolic or hard links. By adding this feature directly to the server source, you can use process memory to track the search results rather than objects on the hard disk. - Re-write all file-handling routines to reference the original path of a file when operating on items within the search results. Cost: - More difficult implementation as it requires more code and must be coded in the same (or language that is understood) by the server software. - If not coded properly, could compromise the stability of the server software. Implementation: *hxd Exec Shell Script Extension Benefits: - Faster development and easier to code. - Can be programmed in any language that the underlying operating system can execute. For example, bourne shell, Perl, Python, C, etc. - Much more difficult to compromise the stability of the server software if improperly implemented. - Can take advantage of hundreds of system utilities to perform standard tasks such as searching for a file (for example, under most UNIX/Linux environments you can use find(1) or locate(1)). Cost: - Resource-intensive. Each time hxd runs an Exec-Script, it has to fork off a new process. This is an expensive operation when it comes to CPU time and memory that has to be allocated. - Possibly slower implementation. Depending on which language you write your extension in, it may run much slower than the first implementation method. - Disk access. This method creates links on the hard drive which point to the originals matching any search criteria. This will increase the amount of time that is required to display the results to the user. - Little or no event-driven tasks. Shell extensions to hxd do not have access to user-events or other internals. Therefore, to implement automatic cleanup or other event-driven tasks, you may have to create a cron-job (time-driven task run by the system). 4. Detailed Information Depending on which method you use to implement this feature in your software, there will be different challenges to overcome. In this section, I will outline the different challenges that you will have to meet. Last, I will discuss in detail some of the solutions. Implementation: Both Issues/Challenges: - How do you deal with different users performing concurrent searches and display the results to that user and only that user? Implementation: Modification of Server-Software Code Issues/Challenges: - None Implementation: *hxd Exec Shell Script Extension Issues/Challenges: - If you try to delete or move a file/folder, the link will be acted upon rather than the original. Solutions for concurrent and separate user-specific searches: In order to maintain separate search results for each user, you will have need to use the unique user-identifier (UID) that is provided to each user by the server at login. In the implementation where you code this feature directly into the server software, you could add either maintain a global array of search results tied to each user's UID, or you could add a data structure to the unique object representing that user. When you receive a request that requires manipulation or retrieving of the search results, you need only query the UID of the user making the request. Thus, you know which data to return to the user. In the implementation where you write an *hxd extension, you first must start with *hxd version that exports the $UID variable into the executables environment. You can then store the search results as links in a temporary directory specific to that user. For example, if a user with UID 25 performs a "/search" command, you will likely store the links to the matching items in "tmp/search_results.25". However, you now need to enable a way for the user to obtain the search results from the [inaccessible] temporary directory. This may require modification of the server software, because you will need to output the contents of a single, specific directory based on the user's UID. One way to accomplish this with minimal customization to the server software would be to rewrite the path translation routines in hxd to simply translate all requests for "/Search Results" to the local path "tmp/search_results.UID". Solutions for deleting/moving links instead of targets: I am open to suggestions on this one. 5. Conclusion With little work and proper implementation, this feature can add the file searching features that we have been looking for in Hotline for a long time. This feature does not require any modification to any client to work. Once implemented, anybody who logs into a supporting Hotline server will be able to search faster and easier than ever before.