Tue Apr 11 00:21:48 2000 Geoff Hutchison * htsearch/htsearch.cc (setupWords): Does not apply fuzzy algorithms to phrase queries. This helps prevent the infinite loops described on the mailing list. * htcommon/conf_parser.yxx (list): Add conditions for lists starting with string-number, number-string, and number-number. * htcommon/conf_parser.cxx: Regenerate using bison. * htdoc/RELEASE.html: Update release notes for recent bug fixes and likely release date for 3.2.0b2. * htdoc/main.html: Add a blurb about the 3.2.0b2 release. * htdoc/*.html: Remove author notes in the footer as requested by Andrew. To balance it out, the copyright notice at the top links to THANKS.html. Sun Apr 9 15:21:12 2000 Geoff Hutchison * htcommon/conf_parser.yxx (list): Fix problem with build_select_lists--parser didn't support lists including numbers. * htcommon/conf_parser.cxx: Regenerate using bison. Sun Apr 9 12:53:02 2000 Geoff Hutchison * htdoc/RELEASE.html: Add a first draft of 3.2.0b2 release notes. Sun Apr 9 12:31:13 2000 Geoff Hutchison * httools/Makefile.am, httools/Makefile.in: Add htload to compilation list. * htcommon/DocumentDB.h: Add optional verbose options to DumpDB and LoadDB. * htcommon/DocumentDB.cc (LoadDB): Implement loading and parsing an ASCII version of the document database. Records on disk will replace any matching records in the db. (DumpDB): Add all fields in the DocumentRef to ensure the entire database is written out. * htcommon/DocumentRef.h: Add new method for setting DocStatus from an int type. * htcommon/DocumentRef.cc (DocStatus): Set it using a switch statement. (It's not pretty, but it works.) * httools/htload.cc: New file. Loads in ASCII versions of the databases, replacing existing records if found. * httools/htdump.cc: Pass verbose flags to DumpDB method. Make sure to close the document DB before quitting. * httools/htpurge.cc: Add -u option to specify a URL to purge from the command-line. * httools/htstat.cc: Add -u option to output the list of URLs in the document DB as well. Sat Apr 8 16:35:55 2000 Geoff Hutchison * htcommon/defaults.cc: Change all , , and tags to the HTML-4.0 compliant , , and tags. * installdir/long.html, installdir/header.html, installdir/nomatch.html, installdir/syntax.html, installdir/wrapper.html: Ditto. * htdoc/*.html: Ditto. (Don't you just love sed?) * htsearch/TemplateList.cc (createFromString): Ditto. * htdoc/htpurge.html, htdoc/htdump.html, htdoc/htload.html, htdoc/htstat.html: New files documenting usage of httools programs. * htdoc/contents.html: Add links to above. * htdoc/htdig.html: Update table with -t format to match htdump. Fri Apr 7 00:30:01 2000 Geoff Hutchison * README: Update to mention 3.2.0b2 and use correct copyright. (It is 2000 after all!) * htdoc/FAQ.html, htdoc/where.html, htdoc/uses.html, htdoc/isp.html: Update with most recent versions from maindocs. * htdoc/RELEASE.html: Add release notes for 3.1.5 to the top. (It's out of version ordering, but it is in correct chronological order.) Fri Apr 7 00:11:29 2000 Geoff Hutchison * httools/htpurge.cc (main): Read in URLs from STDIN for purging, one per line. Pass them along to purgeDocs for removal. Also, make discard_list into a local variable and pass it from purgeDocs to purgeWords. (purgeDocs): Accept a hash of URLs to delete (user input) and return the list of doc IDs deleted. (usage): Note the - option to read in URLs to be deleted from STDIN. Thu Apr 6 00:10:23 2000 Geoff Hutchison * htdig/Retriever.cc (got_redirect): Allow the redirect to accept relative redirects instead of just full URLs. Wed Apr 5 15:07:52 2000 Gilles Detillieux * htsearch/Display.cc: Added #if test to make sure DBL_MAX is defined on Solaris, as reported by Terry Luedtke. Tue Apr 4 12:46:37 2000 Gilles Detillieux * contrib/doc2html/*: Added parser submitted by D.J.Adams@soton.ac.uk Mon Apr 3 13:48:59 2000 Gilles Detillieux * htcommon/defaults.cc: Fix error in description of new attribute plural_suffix. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Fri Mar 31 21:48:02 2000 Geoff Hutchison * configure.in, configure: Add test using AC_TRY_RUN to compile against the htlib/regex.c and attempt to compile a regexp. This should allow us to find out if the included regex code causes problems. * acconfig.h: Add HAVE_BROKEN_REGEX as a result of the configure script to conditionally include the appropriate regex.h file. * include/htconfig.h.in: Regenerate using autoheader. * htlib/regex.c: Move #include "htconfig.h" inside HAVE_CONFIG_H tests. This file is only created when this is true anyway. This prevents problems with the configure test. * htlib/HtRegex.h, htfuzzy/EndingsDB.cc: Use HAVE_BROKEN_REGEX switch to use the system include instead of the local include where appropriate. * htlib/Makefile.am, htlib/Makefile.in: Only compile regex.lo if the configure script added it to LIBOBJS. Thu Mar 30 22:41:38 2000 Geoff Hutchison * htcommon/URL.cc (normalizePath): Remove Gilles's loop to add back ../ components to a path that would go above the top level. Now we simply discard them. Both are allowed under the RFC, but this should have fewer "surprises." Tue Mar 28 21:57:49 2000 Geoff Hutchison * htnet/Connection.cc (Read_Partial): Fix bug reported by Valdas where a zero value returned by select would result in an infinite loop. * htcommon/defaults.cc: Add new attribute plural_suffix to set the language-dependent suffix for PLURAL_MATCHES contributed by Jesse. * htsearch/Display.cc (setVariables): Use it. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Mon Mar 27 22:28:20 2000 Geoff Hutchison * htcommon/DocumentRef.cc (Deserialize): Add back stub for DOC_IMAGESIZE to prevent decoding errors. This just throws away that field. * htcommon/HtSGMLCodec.h (class HtSGMLCodec): Differentiate between codec used for &foo; and numeric form &#nnn; Make sure encoding goes through both but decoding only goes through the preferred text form. * htcommon/HtSGMLCodec.cc (HtSGMLCodec): When constructing the private HtWordCodec objects, create separate lists for the number and text codecs. Mon Mar 27 21:25:27 2000 Geoff Hutchison * htsearch/HtURLSeedScore.cc (ScoreAdjustItem): Change to use HtRegex for flexibility and to get around const char * -> char * problems. * htsearch/SplitMatches.cc (MatchArea): Ditto. * htsearch/Makefile.am, htsearch/Makefile.in: Add SplitMatches.cc and HtURLSeedScore.cc to compilation list! Mon Mar 27 21:03:12 2000 Hans-Peter Nilsson * htcommon/defaults.cc (defaults): Add default for search_results_order, url_seed_score. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerated using cf_generate.pl. * htlib/List.h (List): New method AppendList. * htlib/List.cc (List::AppendList): Implement it. * htsearch/SplitMatches.h, htsearch/SplitMatches.cc: New. * htsearch/HtURLSeedScore.cc, HtURLSeedScore.h: New. * htsearch/Display.h (class Display: Add member minScore. Change maxScore type to double. * htsearch/Display.cc: Include SplitMatches.h and HtURLSeedScore.h (ctor): Initialize minScore, change init value for maxScore to -DBL_MAX. (buildMatchList): Use a SplitMatches to hold search results and interate over its parts when sorting scores. Ignore Count() of matches when setting minScore and maxScore. Use an URLSeedScore to adjust the score after other calculations. Calculate minScore. Correct maxScore adjustment for change to double. (displayMatch): Use minScore in calculation of score to adjust for negative scores. (sort): Calculation of maxScore moved to buildMatchList. Mon Mar 27 20:22:24 2000 Geoff Hutchison * htcommon/DocumentRef.h, htcommon/DocumentRef.cc: Remove DocImageSize field since it is not used anywhere and is never updated. * htdig/Retriever.h (class Retriever): Remove references to Images class. * htcommon/DocumentDB.cc (DumpDB): Ignore DocImageSize field. * htdig/Makefile.am, htdig/Makefile.in: Remove Images.cc since this is no longer used. * htdig/Plaintext.cc: Do not insert SGML equivalents into the excerpt, these are decoded by HtSGMLCodec automatically. Sat Mar 25 21:58:36 2000 Geoff Hutchison * htdoc/cf_generate.pl (html_escape): Changed and tags to HTML 4.0 and tags. Sat Mar 25 17:23:46 2000 Geoff Hutchison * htdb/Makefile.am, htdb/Makefile.in: Change the names of the htdb utility programs to escape name conflicts with httool programs. * htdb/htdb_load.cc: Rename htload.cc to escape name conflict and more closely match orignal db_load program name. * htdb/htdb_dump.cc, htdb/htdb_stat.cc: Ditto. * htfuzzy/Prefix.cc (getWords): Add code to "weed out" duplicates returned from WordList::Prefix. We only want to add unique words to the search list. Fri Mar 24 22:33:20 2000 Geoff Hutchison * htdig/Document.cc (Document): Fix bug reported by Mentos Hoffman, contributed by Atlee Gordy . Mon Mar 20 23:14:26 2000 Geoff Hutchison * htcommon/DocumentDB.cc (Delete): Fix bug reported by Valdas where duplicate document records could "sneak in" because the doc_index entry was removed incorrectly. Mon Mar 20 19:08:14 2000 Geoff Hutchison * htcommon/defaults.cc: Added block field and added appropriate blocks. * htlib/Configuration.h (struct ConfigDefaults): Add block field. * htdoc/cf_generate.pl: Parse the new block field. * htdoc/cf_byname.html, htdoc/cf_byprog.html, htdoc/attrs.html: Regenerate using above. * htcommon/DocumentDB.cc (DumpDB): Make sure we decompress the DocHead field before we write it to disk! * httools/htdump.cc, httools/htstat.cc: Call WordContext::Initialize() before doing any htword calls. Mon Mar 20 14:10:30 2000 Geoff Hutchison * httools/htpurge.cc: Whoops! Left some references to htmerge in the error messages and usage message. * httools/htstat.cc: New program. Simply spits up the total number of documents, words and unique words in the databases. * httools/htdump.cc: New program. Simply dumps the contents of the document DB and the word DB to doc_list and word_dump files respectively. Also has flags -w and -d to pick one or the other. * httools/Makefile.am, httools/Makefile.in: Add htdump and htstat programs to compilation list. * htcommon/DocumentDB.cc (DumpDB): Change name of CreateSearchDB and add fields for DocBackLinks, DocSig, DocHopCount, DocEmail, DocNotification, and DocSubject. This should now export every portion of the document DB. * htcommon/DocumentDB.h: Change name of CreateSearchDB and add stub for LoadDB, to be written shortly. * htdig/htdig.cc: Call DumpDB instead of CreateSearchDB when creating an ASCII version of the DB. Sat Mar 18 22:57:02 2000 Geoff Hutchison * httools/Makefile.am, httools/Makefile.in: New directory for useful database utilities. * httools/htnotify.cc: Moved htnotify to httools directory. * httools/htpurge.cc: New program--currently just purges documents (and corresponding words) in the databases. Will shortly also allow deletion of specified URLs. * Makefile.am, configure.in: Remove htnotify directory in favor of httools directory. * configure: Regenerate using autoconf. * Makefile.in: Regenerate using automake --foreign. Fri Mar 17 16:47:37 2000 Gilles Detillieux * htsearch/Display.cc (excerpt, hilight): Correctly handle case where there is no pattern to highlight. * htsearch/htsearch.cc (addRequiredWords), htcommon/defaults.cc: Add any_keywords attribute, to OR keywords rather than ANDing, fix addRequiredWords not to mess up expression when there are no search words, but required words are given. * htdoc/hts_form.html: Mention new attribute, add links to all mentioned attributes. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Fri Mar 17 15:48:12 2000 Gilles Detillieux * htfuzzy/Accents.cc (generateKey): Truncate words to maximum_word_length, for consistency with what's found in word DB. Fri Mar 17 10:56:17 2000 Gilles Detillieux * htdig/HTML.cc (do_tag): Use case insensitive parsing of META robots tag content. * htlib/String.cc (uppercase): Fix misplaced cast for islower(). Mon Mar 6 17:31:37 2000 Gilles Detillieux * htsearch/htsearch.cc (setupWords): Don't allow comma as string list separator, as it can be a decimal point in some locales. Mon Mar 06 00:58:00 2000 Loic Dachary * db/mp/mp_bh.c (__memp_bhfree): always free the chain, if any. The bh is reset to null after free and we loose the pointer anyway, finally filling the pool with it. * db/mp/mp_cmpr.c (__memp_cmpr_write): i < CMPR_MAX - 1 instead of i < CMPR_MAX otherwise go beyond array limits. This fixes a major problem when handling large files. Sat Mar 04 19:41:49 2000 Loic Dachary * db/mp/mp_cmpr.c (__memp_cmpr_free_chain): clear BH_CMPR flag. Was causing core dumps, thanks to Peter Marelas maral@phase-one.com.au for providing a simple case to reproduce the error. Fri Mar 3 11:32:34 2000 CEST Gabriele Bartolini * Fixed bugs regarding yesterday's changes. Even Leonardo da Vinci used to commit errors, so ... Fri Mar 3 11:25:42 2000 CEST Gabriele Bartolini * testnet.cc: added the -r and -w options in order to set how many times it retries to re-connect after a timeout occurs, and how long it should wait after it. Thu Mar 2 18:45:15 2000 CEST Gabriele Bartolini * htnet/Connection.*: management of wait_time and number of retries after a timeout occurs. * htnet/Transport.*: Management of connection attributes above. * htdig/Server.*: Set members for managing timeout retries taken from the configuration file ("timeout", "tcp_max_retries", "tcp_wait_time"). * htdig/Document.cc: Added the chance to configure on a server basis "persistent_connections", "head_before_get", "timeout", "tcp_max_retries", "tcp_wait_time". Changed Retrieve method accepting now a server object pointer: Retrieve (server*, HtDateTime). * htdig/Retriever.cc: Added the chance to configure on a server basis "max_connection_requests" attribute. * htcommon/defaults.cc: Added "tcp_max_retries", "tcp_wait_time" -- Need to be go over by someone who speaks english better than me. Not a hard work !!! ;-) Wed Mar 1 17:01:09 2000 Gilles Detillieux * htsearch/Display.cc (excerpt, hilight): move SGML encoding into hilight() function, because when it's done earlier it breaks highlighting of accented characters. Wed Mar 1 16:02:49 2000 Gilles Detillieux * htfuzzy/htfuzzy.cc (main): Correctly test return value on Open() of word database, include db name in error message if Open() fails, do a WordContext::Initialize() before we need htword functions. (Obviously I'm the first to test htfuzzy in 3.2!) * htfuzzy/Accents.cc (generateKey): cast characters to unsigned char before using as array subscripts. Wed Mar 1 13:27:26 2000 Gilles Detillieux * htcommon/defaults.cc: Added accents_db attribute, mentioned accents algorithm in search_algorithms section. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. * installdir/htdig.conf: Added mentions of accents, speling & substring, fixed a couple typos in comments. * htdoc/htfuzzy.html: Added blurb on accents algorithm. * htdoc/require.html: Added mentions of accents, speling, substring, prefix & regex. * htdoc/config.html: Updated with sample of latest htdig.conf and installdir/*.html, added blurb on wrapper.html. Wed Mar 1 00:30:19 2000 Geoff Hutchison * configure.in, configure: Add test for FD_SET_T, the second (also third and fourth) argument in calls to select(). Should solve PR#739. * acconfig.h, include/htconfig.h.in: Add declaration for FD_SET_T. * htnet/Connection.cc (ReadPartial): Change declaration of fds to use FD_SET_T define set by the configure script. Tue Feb 29 23:11:49 2000 Geoff Hutchison * htlib/DB2_db.cc (Error): Simply fprint the error message on stderr. This is not a method since the db.h interface expects a C function. (db_init): Don't set db_errfile, instead set errcall to point to the new Error function. Tue Feb 29 15:09:41 2000 Gilles Detillieux * htfuzzy/Accents.h, htfuzzy/Accents.cc: Adapted writeDB() for 3.2. Tue Feb 29 14:29:37 2000 Gilles Detillieux * htfuzzy/Accents.h, htfuzzy/Accents.cc: Added these, as contributed by Robert Marchand, to implement accents fuzzy match. Adapted to 3.2. * htfuzzy/Fuzzy.cc, htfuzzy/htfuzzy.cc, htfuzzy/Makefile.am, htfuzzy/Makefile.in: Added in accents algorithm, as for soundex. Tue Feb 29 11:31:53 2000 Loic Dachary * test/testnet.cc (Listen): Add -b port to listen to a specific port. This is to test connect timeout conditions. * htnet/Connection.cc (Connect): Added SIGALRM signal handler, Connect() always allow EINTR to occur. Mon Feb 28 15:32:46 2000 Loic Dachary * htword/WordKey.h (class WordKey): explicitly add inline keyword for all inline functions. Mon Feb 28 13:10:34 2000 Loic Dachary * htword/WordKey.h (class WordKey): nfields data member caches result of NFields() method. * htword/WordDBPage.h (class WordDBPage): nfields data member caches result of WordKey::NFields() method. * acinclude.m4 (APACHE): check in lib/apache for modules Sat Feb 26 22:05:03 2000 Geoff Hutchison * htsearch/Collection.h, htsearch/Collection.cc: New files contributed by Rajendra Inamdar . * htsearch/Makefile.am, htsearch/Makefile.in: Compile them. * htcommon/defaults.cc: Add new collection_names attribute as described by Rajendra. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. * htsearch/Display.h, htsearch/Display.cc: Loop through collections as we are assembling results. (buildMatchList): Use 1.0 as minimum score and take log(score) as the final score. This requires an increase in magnitude in weight to correspond to a factor of increase in score. * htsearch/DocMatch.h, htsearch/DocMatch.cc: Keep track of the collection we're in. * htsearch/ResultMatch.h: Ditto. * htsearch/htsearch.h, htsearch/htsearch.cc: Wrap results in collections. * htsearch/parser.h, htsearch/parser.cc: Set the collection for the results--we use this to get to the appropriate word DB. (score): Divide word weights by word frequency to calibrate for expected Zipf's law. Rare words should count more. Fri Feb 25 11:19:47 2000 Gilles Detillieux * htcommon/defaults.cc (maximum_pages): Describe new bahaviour (as of 3.1.4), where this limits total matches shown. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Thu Feb 24 14:43:06 2000 Gilles Detillieux * htnet/HtFile.cc (Request): Fix silly typo. * htlib/DB2_db.cc: Remove include of malloc.h, as it causes problems on some systems (e.g. Mac OS X), and all we need should be in stdlib.h. Thu Feb 24 13:11:15 2000 Gilles Detillieux * htnet/HtFile.cc (Request): Don't append more than _max_document_size bytes to _contents string, set _content_length to size returned by stat(). * htnet/HtHTTP.cc (HTTPRequest): Extra tests in case Content-Length not given for non-chunked input, and not to close persistent connection when chunked input exceeds _max_document_size. (ReadChunkedBody): Don't append more than _max_document_size bytes to _contents string. Thu Feb 24 11:40:24 2000 Gilles Detillieux * htdig/HTML.cc (do_tag): Fix handling of img alt text to be consistent with body text, rather than keywords. * htdig/Retriever.cc (ctor): Treat alt text as plain text, until it has its own FLAG and factor. Thu Feb 24 11:16:37 2000 Gilles Detillieux * htcommon/defaults.cc (version): Moved example over to correct field. (defaults[] terminator): Padded zeros to new number of fields. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Thu Feb 24 19:08:41 2000 Loic Dachary * htmerge/words.cc: only display Word in verbose message instead of complete key if verbosity < 3. Thu Feb 24 10:43:12 2000 Gilles Detillieux * htcommon/defaults.cc (external_protocols, external_parser): Swapped these two entries to put them in alphabetical order. (star_blank): Fixed old typo (incorrect reference to image_star). * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Wed Feb 23 16:53:40 2000 Gilles Detillieux * htcommon/defaults.cc (backlink_factor, external_parser, local_default_doc, local_urls, local_urls_only, local_user_urls): Add some updates from 3.1.5's attrs.html. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Wed Feb 23 15:11:51 2000 Gilles Detillieux [ Improve htsearch's HTML 4.0 compliance ] * htsearch/TemplateList.cc (createFromString): Use file name rather than internal name to select builtin-* templates, use $&(TITLE) and $&(URL) in templates and quote HTML tag parameters. * installdir/long.html, installdir/short.html: Use $&(TITLE) and $&(URL) in templates and quote HTML tag parameters. * htsearch/Display.cc (setVariables): quote all HTML tag parameters in generated select lists. * installdir/footer.html, installdir/header.html, installdir/nomatch.html, installdir/search.html, installdir/syntax.html, installdir/wrapper.html: Use $&(var) where appropriate, and quote HTML tag parameters. * installdir/htdig.conf: quote all HTML tag parameters. Wed Feb 23 13:40:27 2000 Gilles Detillieux * htcommon/URL.h (encodeURL): Change list of valid characters to include only unreserved ones. * htcommon/cgi.cc (init): Allow "&" and ";" as input param. separators. * htsearch/Display.cc (createURL): Encode each parameter separately, using new unreserved list, before piecing together query string, to allow characters like "?=&" within parameters to be encoded. Wed Feb 23 13:22:29 2000 Gilles Detillieux * htlib/URL.cc (ServerAlias): Fix server_aliases processing to prevent infinite loop (as for local_urls in PR#688). Wed Feb 23 12:49:52 2000 Gilles Detillieux * htlib/HtDateTime.h, htlib/HtDateTime.cc: change Httimegm() method to HtTimeGM(), to avoid conflict with Httimegm() C function, so we don't need "::" override, for Mac OS X. * htlib/htString.h, htlib/String.cc: change write() method to Write(), to avoid conflict with write() function, so we don't need "::" override, for Mac OS X. Wed Feb 23 12:17:46 2000 Gilles Detillieux * htlib/Configuration.cc(Read): Fixed to allow final line without terminating newline character, rather than ignoring it. Wed Feb 23 12:01:01 2000 Gilles Detillieux * htdig/Retriever.cc (GetLocal, GetLocalUser): Add URL-decoding enhancements to local_urls, local_default_urls & local_default_doc, to allow hex encoding of special characters. Wed Feb 23 19:14:29 2000 Loic Dachary * htcommon/conf_parser.cxx: regenerated from conf_parser.yxx Wed Feb 23 19:04:16 2000 Loic Dachary * test/test_functions.in: inconditionaly remove existing test/var directory before runing tests to prevent accidents. * htcommon/URL.cc (URL): fixed String->char warning * htcommon/defaults.cc (wordlist_compress): defaults to true Tue Feb 22 17:09:10 2000 Gilles Detillieux * htdig/HTML.cc(parse, do_tag): Fix handling of ... text and parsing of words in meta tags, to to proper word separation. * htlib/HtWordType.h, htlib/HtWordType.cc: Add HtWordToken() function, to replace strtok() in HTML parser. Tue Feb 22 16:21:25 2000 Gilles Detillieux * htlib/URL.cc (ctor, normalizePath): Fix PR#779, to handle relative URLs correctly when there's a trailing ".." or leading "//". Tue Feb 22 14:09:26 2000 Gilles Detillieux * htdig/Document.cc (RetrieveLocal): Handle common extensions for text/plain, application/pdf & application/postscript. Mon Feb 21 17:25:21 2000 Gilles Detillieux * contrib/htdig-3.2.0.spec: Fixed %post script to add more descriptive entries in htdig.conf, made cron script a config file, updated to 3.2.0b2. * contrib/conv_doc.pl, contrib/parse_doc.pl: Added comments to show Warren Jones's updates in change history. Mon Feb 21 17:09:13 2000 Gilles Detillieux * htcommon/HtConfiguration.h, htcommon/conf_parser.yxx, htlib/Configuration.h, htlib/Configuration.cc: split Add() method into Add() and AddParsed(), so that only config attributes get parsed. Use AddParsed() only in Read() and Defaults(). Fri Feb 18 22:50:54 2000 Geoff Hutchison * htnet/Connection.h, htnet/Connection.cc: Renamed methods with capitals to remove the need to use ::-escaped library calls. * htnet/Transport.h, htnet/Transport.cc, htnet/HtHTTP.cc, htdig/Images.cc: Fix code using Connection to use the newly capitalized methods. Fri Feb 18 14:40:50 2000 Loic Dachary * test/conf/access.conf.in: removed cookies. Not used and some httpd are not compiled with usertrack. Wed Feb 16 12:15:08 2000 Vadim Chekan * htcommon/Makefile.am replaced conf.tab.cc.h by conf_parser.h in noinst_HEADERS * htcommon/conf_parser.yxx,conf_parser.lxx,HtConfiguration.cc, HtConfiguration.h: added copyright and Id: * htcommon/cgi.cc(init): fixed bug: array must be free by delete [] buf, not just delete buf; Tue Feb 15 23:16:14 2000 Geoff Hutchison * htnet/HtHTTP.cc (isParsable): Remove application/pdf as a default type--it is now handled through the ExternalParser interface if at all. * htcommon/defaults.cc: Remove pdf_parser attribute. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. * htdig/Document.cc (getParsable): Remove PDF once and for all (hopefully). * htdig/ExternalParser.cc (parse): Ditto. * configure.in: Remove check for PDF_PARSER. * configure: Regenerate using autoconf * htdig/Makefile.am: Remove PDF.cc and PDF.h. * Makefile.in, */Makefile.in: Regenerate using automake --foreign Tue Feb 15 12:02:39 EET 2000 Vadim Chekan * htcommon/HtConfiguration.cc,HtConfiguration.h: fixed bug discovered by Gilles. HtConfiguration was able to get info only from "url" and "server" block. * htcommon/conf_parser.yxx: deleted 1st parameter for new char[], lefted when realloc was replaced by new char[]. Removed a few unused variable declaration. * htcommon/Makefile.am: added -d flag to bison to generate conf_parser.h template from conf_parser.yxx; conf_lexer.lxx uses #include conf_parser.h; conf.tab.cc.h removed. Sun Feb 13 21:19:04 2000 Geoff Hutchison * htcommon/defaults.cc: Get rid of uncoded_db_compatible since the current DB format has clearly broken backwards compatibility. * htsearch/Display.cc (Display), htnotify/htnotify.cc (main), htmerge/docs.cc (convertDocs), htmerge/db.cc (mergeDB), htdig/htdig.cc (main): Remove call to DocumentDB::setCompatibility(). * htcommon/DocumentDB.h (class DocumentDB): Remove setCompatibility and related private variable. * htcommon/DocumentDB.cc ([], Delete): Don't bother checking for an unencoded URL, at this point all URLs will be encoded using HtURLCodec. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Sat Feb 12 21:29:20 2000 Geoff Hutchison * htcommon/HtSGMLCodec.cc (HtSGMLCodec): Always translate " & < and > * htcommon/defaults.cc: Remove translate_* and word_list attributes since they're now no longer used. * htdig/PDF.cc (parseNonTextLine): Fix bogus escape sequences around Title parsing. Fixes PR#740. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Fri Feb 11 11:41:36 2000 Loic Dachary * htlib/Makefile.am: removed CFLAGS=-g (use make CXXFLAGS=-g all instead). * htdoc/install.html: specify header/lib install directory now is prefix/include/htdig and prefix/lib/htdig. * Makefile.am (distclean-local): use TESTDIR instead of deprecated HTDIGDIRS. * */Makefile.am: install libraries in prefix/lib/htdig and includes in prefix/include/htdig. Just prepend pkg in front of automake targets. * include/Makefile.am: install htconfig.h Thu Feb 10 23:18:37 2000 Loic Dachary * Connection.cc (Connection): set retry_value to 1 instead of 0 as suggested by Geoff. Thu Feb 10 17:36:09 2000 Loic Dachary * htdig/Document.cc: fix (String)->(char*) conversion warnings. * htword/WordList.cc: kill Collect(WordSearchDescription) which was useless and error prone. * htword/WordDB.h (WordDBCursor::Get): small performance improvement by copying values only if key found. * htword/WordDB.h,WordList.cc: fix reference counting bug when using Override (+1 even if entry existed). Turn WordDB.h return values to be std Berkeley DB fashion instead of the mixture with OK/NOTOK that was a stupid idea. This allows to detect Put errors and handle them properly to fix the Override bug without performance loss. * test/conf/httpd.conf.in: comment out loading of mod_rewrite since not everyone has it. Thu Feb 10 00:26:02 2000 Geoff Hutchison * htcommon/defaults.cc: Add new attribute "nph" to send out non-parsed headers for servers that do not supply HTTP headers on CGI output (e.g. IIS). * htsearch/Display.cc (display): If nph is set, send out HTTP OK header as suggested by Matthew Daniel (displaySyntaxError): Ditto. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate from current defaults.cc file. Thu Feb 10 00:21:58 2000 Geoff Hutchison * htdig/HTML.cc (do_tag): Treat tags as noindex tags, much like as suggested by Torsten. Thu Feb 10 00:02:41 2000 Geoff Hutchison * .version: Bump for 3.2.0b2. * htcommon/defaults.cc: Add category fields for each attribute. Though these are currently unused, they could allow the documentation to be split into multiple files based on logical categories and subcategories. Wed Feb 9 23:52:55 2000 Geoff Hutchison * htnet/Connection.cc (connect): Add alarm(timeout) ... alarm(0) around ::connect() call to ensure this does timeout as appropriate as suggested by Russ Lentini to resolve PR#762 (and probably others as well). (connect): Add a retry loop as suggested by Wilhelm Schnell to resolve PR#754. * htnet/HtHTTP.cc (HTTPRequest): Add CloseConnection() when the connection fails on open before returning from the method. Should take care of PR#670 for htdig-3-2-x. Wed Feb 09 17:20:50 2000 Loic Dachary * db/dist/Makefile.in (libhtdb.so): move dependent libraries *after* the list of objects, otherwise it's useless. * htword/WordKey.h (class WordKey): move #if SWIG around to please swig (www.swig.org). * htword/WordList.h (class WordList): allow SWIG to see Walk* functions (#if SWIG). Wed Feb 9 09:21:00 2000 Gilles Detillieux * htdig/Server.cc (robotstxt): apply more rigorous parsing of multiple user-agent fields, and use only the first one. * htlib/HtRegex.cc (set): apply the fix from Valdas Andrulis, to properly compile case_sensitive expressions. Mon Feb 09 09:43:59 2000 CEST Gabriele Bartolini * htnet/HtHTTP.cc: changed "<<" to append() for content_length assignment in ReadChunkedBody() function (as Gilles suggested) Tue Feb 08 10:54:08 2000 Loic Dachary * db/dist/configure.in: Added AC_PREFIX_DEFAULT(/opt/www) so that headers and libraries are installed in the proper directory when no --prefix is given. Tue Feb 08 10:32:48 2000 Loic Dachary * test/t_wordskip: copy $srcdir/skiptest_db.txt to allow running outside the source tree. * configure.in: use '${prefix}/...' instead of "$ac_default_prefix/..." that did not carry the --prefix value. * configure.in: run CHECK_USER and AC_PROG_APACHE if --enable-tests Mon Feb 07 17:40:47 2000 Loic Dachary * htlib/htString.h (last): turn to const Mon Feb 07 14:05:37 2000 CEST Gabriele Bartolini * htnet/HtHTTP.cc: fixed a bug in ReadChunkedBody() function regarding document size assignment (raised by Valdas Andrulis) Sun Feb 06 19:11:05 2000 Loic Dachary * configure.in: Fix inconsistencies between default values shown by ./configure and actual defaults. * htdoc/install.html: change example version 3.1 to 3.2 Commented out warning about libguile. Replace CONFIG variables by configure.in options. Specify default value for each of them. Replace (and move) make depend by automake (distributed Makefiles do not include dependency generation) Added section for running tests. Added section on shared libraries. * configure.in: use AM_CONDITIONAL for --enable-tests * Makefile.am: use automake conditionals for subdir so that make dist knows what to distribution --enable-tests specified or not. * db/Makefile.in: allow make dist to work outside the source tree. Sat Feb 05 18:31:04 2000 Loic Dachary * test/word.cc (SkipTestEntries): The fix of WordList::SkipUselessSequentialWalking actually saves us a few hops when walking lists of words. Fri Feb 04 17:28:32 2000 Loic Dachary * htword/WordKey.cc,WordReference.cc,WordRecord.cc (Print): use cerr instead of cout for immediate printing under debugger. Thu Feb 3 16:06:45 2000 Gilles Detillieux * htdig/Document.cc (RetrieveLocal): fix bug that prevented local filesystem digging, because max_doc_size was initialized to 0. Now sets it to max_doc_size for current url. Thu Feb 3 12:36:56 2000 Gilles Detillieux * installdir/Makefile.{am,in}: install mime.types as mime.types, not as htdig.conf. * htfuzzy/EndingsDB.cc (createDB): fix code to use MV macro in system() command, not hard-coded "MV" string literal, and use get() on config objects to avoid passing String objects to form(). Wed Feb 2 19:44:33 2000 Geoff Hutchison * htlib/HtDateTime.cc (SetRFC1123): Strip off weekday, if present and use LOOSE format. (SetRFC850): Ditto. * configure.in, configure: Add configure check for "mv." * htfuzzy/Makefile.am: Use it. * */Makefile.in: Regenerate using automake. * htfuzzy/EndingsDB.cc (createDB): Use the detected mv, or whatever is in the path to move the endings DB when they're finished. Wed Feb 2 15:49:14 2000 Gilles Detillieux * htdig/Document.cc (RetrieveLocal), htdig/Retriever.cc (GetLocal): Fix compilation errors. Oops! Wed Feb 2 13:53:27 2000 Gilles Detillieux * htdig/Retriever.cc (IsValidURL): fix problem with valid_extensions matching failure when URL parameters follow extension. Wed Feb 2 13:29:48 2000 Gilles Detillieux * htlib/QuotedStringList.cc (Create): fix PR#743, where quoted string lists didn't allow embedded quotes of opposite sort in strings (e.g. "'" or '"'), and fix to avoid overrunning end of string if it ends with backslash. Wed Feb 2 13:23:16 2000 Gilles Detillieux * htdig/HTML.cc (ctor, parse, do_tag), htcommon/defaults.cc: Add max_keywords attribute to limit meta keyword spamming. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Wed Feb 2 12:57:40 2000 Gilles Detillieux * htdig/Document.cc (RetrieveLocal), htdig/Document.h, htdig/Retriever.cc (Initial, parse_url, GetLocal, GetLocalUser, IsLocalURL, got_href, got_redirect), htdig/Retriever.h, htdig/Server.cc (ctor), htdig/Server.h: Add in Paul Henson's enhancements to local_urls, local_default_urls & local_default_doc. * htcommon/defaults.cc: Document these. Wed Feb 02 10:14:57 2000 Loic Dachary * htword/WordKeyInfo.h,WordKey.{cc,h}: fix overflow bug when 32 bits. For that purpose implement Outbound/Overflow/Underflow methods in WordKey, MaxValue in WordKey/WordKeyInfo. (WordKey::SetToFollowing) was FUBAR : overflow of field1 tested with number of bits in next field, do not handle overflow, Re-implemented. (WordKey::Set) Change atoi to strtoul. (WordList::SkipUselessSequentialWalking) was much to fucked up to explain. Re-implement (WordKey::Diff) Added as a support function of SkipUselessSequentialWalking. implement consistent verbosity. * htword/WordList.cc (operator >>): explicit error message when insert failed, with line number. Wed Feb 2 00:11:03 2000 Geoff Hutchison * htdoc/RELEASE.html: Finish up with notes on all significant new attributes. * htdoc/FAQ.html, htdoc/where.html: Mention new 3.2.0b1 release as a beta. * contrib/README: Update to mention new scripts. * installdir/mime.types: Add default Apache mime.types file for systems that do not already have one. * installdir/Makefile.am: Make sure it is installed by default. * installdir/Makefile.in: Regenerate using automake. * htcommon/defaults.cc: Add documentation for mime_types attribute, remove currently unused image_alt_factor, and add documentation for external_protocols. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Regenerate using cf_generate.pl. Tue Feb 1 10:24:19 2000 Gilles Detillieux * htsearch/parser.cc (score): fix up score calculations for correctness and efficiency. Mon Jan 31 16:29:20 2000 Marcel Bosc * htword/WordBitCompress.cc: fixed endian bug in compression Sat Jan 29 21:14:03 2000 Geoff Hutchison * htsearch/parser.cc (score): Change config.Value (which returns int) to config.Double to preserve accuracy of attributes. * htcommon/defaults.cc: Updated documentation for attributes now allowing regex, search_algorithms (for new fuzzy) and added documentation for the overlooked remove_unretrieved_urls. * htdoc/*.html: Updated copyright notice for 2000, changed footer to use CVS's magic Date keyword. Regenerated documentation from defaults changes. Sat Jan 29 16:32:08 2000 Geoff Hutchison * contrib/htdig-3.1.4.spec, contrib/htdig-3.1.4-conf.patch: Remove these since they don't apply to the 3.2.x releases. * htfuzzy/Synonym.cc (openIndex): Change database format from DB_BTREE to DB_HASH--no reason for the synonym database to be a btree. This was probably overlooked when I switched the rest of the fuzzy databases over to DB_HASH. Sat Jan 29 05:34:26 2000 Loic Dachary * htword/WordKey.h (UnpackNumber): Very nasty bug. Optimization dated Dec 29 broke endianess on Solaris. Restore previous version. Fri Jan 28 18:17:08 2000 Geoff Hutchison * htlib/Configuration.h (struct ConfigDefaults): Add version and category fields for more accurate documentation. * htcommon/defaults.cc: Add blank category fields and start filling in version field. Killed modification_time_is_now_attribute. * htdig/Document.cc (Document): Kill attribute modification_time_is_now since it can cause more harm than good. * htnet/HtHTTP.cc (ParseHeader): Ditto. * htdoc/cf_generate.pl: Added support for new version and category fields. Currently category does nothing, but it could split the documentation into categories. Sat Jan 29 01:37:45 2000 Loic Dachary * .version: remove the trailing -dev Thu Jan 27 12:22:57 2000 Loic Dachary * htword/WordList.cc: cdebug replaced by cerr. replace lverbose by verbose > 2. Remove shutup. (WordList): monitor = 0 (Open): create monitor only if wordlist_monitor = true (Close): delete monitor if set, delete compressor if set * htword/WordDBCompress.cc,WordList.cc: only activate monitoring code if monitor is set. No interaction with the monitor is therefore possible if wordlist_monitor is false. * htword/WordMonitor.cc: remove useless test of wordlist_monitor (done by WordList now). * htword/WordDBCompress.cc (TestCompress): remove redundant debuglevel argument. * htword/WordDBCompress.cc (WordDBCompress): init cmprInfo to 0 * db/include/db_cxx.h: Add get_mp_cmpr_info method * htword/WordDBCompress.cc (WordDBCompress): set default debug level to 0 * htword/WordDB.h: CmprInfo returns current CmprInfo and non static, overload to set CmprInfo if argument given. * htword/WordDBCompress.h: new CmprInfo() method returns DB_CMPR_INFO object for Berkeley DB database. * htword/WordList.h: add compressor member, kill cmprInfo member. * htword/WordList.cc: Wed Jan 26 20:05:33 2000 Loic Dachary * htword/WordList.cc,htword/WordList.h: get rid of obsolete WordBenchmarking Wed Jan 26 9:14:32 2000 CEST Gabriele Bartolini * htcommon/defaults.cc: added "max_connection_requests". * htdig/Retriever.cc: now manages the attribute above. Tue Jan 25 12:59:01 2000 Loic Dachary * htsearch/Display.cc (setVariables): fixed Display.cc:505: warning: multiline `//' comment Tue Jan 25 8:37:15 2000 CEST Gabriele Bartolini * htdig/Document.h: Added the "HtHTTP *GetHTTPHandler()" method, in order to be able to control an HTTP object outside the Document class. This is useful for the Server class, after the request for robots.txt. We can control the response of a server and check if it supports persistent connections. * htdig/Server.cc: inside the constructor, persistent_connections var is initialized to the configuration parameter value, instead of . Besides, after the request of the robots.txt, it controls and set the attribute for persistent connections, depending on whether the server supports them or not. * htdig/Retriever.cc: modified the Start() method. Now the loop manage HTTP persistent connections "on a server" basis. Indeed, it's a Server object that decides if persisent connections are allowed on that server or not (depending on configuration or capabilities of the remote http server). Mon Jan 24 12:57:45 2000 Gilles Detillieux * htsearch/Display.cc(setVariables): Added double quotes around default selection value in build_select_lists handling. Mon Jan 24 12:37:22 2000 Gilles Detillieux * htsearch/Display.cc(setVariables), htcommon/defaults.cc: Added build_select_lists attribute, to generate selector menus in forms. Added relevant explanations and links to selectors documentation. * htdoc/hts_selectors.html: Added this page to explain this new feature, plus other details on select lists in general. * htdoc/hts_templates.html: Added relevant links to related attributes and selectors documentation. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl Fri Jan 21 18:57:58 EET 2000 Vadim Chekan * htcommon/HtConfiguration.cc: added HtConfiguration::ParseString(char*) method to allow lexer handle "include: ${var}/file.inc" construction * htcommon/conf_lexer.lxx: fixed handling "include: ${var}file.inc" bug. Fri Jan 21 17:04:28 2000 Loic Dachary * htword/WordList.cc (WalkFinish,WalkInit,WalkNextStep): fix typos in error messages and misleading comment. * htword/WordList.h,WordList.cc: move part of WalkInit in WalkRewind so that we have a function to go back to the beginning of possible matches. Wed Jan 19 21:49:57 2000 Geoff Hutchison * htdig/HTML.cc (do_tag): Only add words for META descriptions, keywords, and IMG ALT attributes if doindex is set. * htcommon/DocumentRef.h: Added Reference_obsolete for documents that should be removed (but haven't). * htdig/Retriever.cc (parse_url): Flag documents that have been modified as Reference_obsolete and update the database. Flag all documents with various errors as something other than Reference_normal, as appropriate--these probably should be pruned. * htdig/Retriever.h: Get rid of GetRef() method--it's only used once! * htsearch/Display.cc (display): Don't show DocumentRefs with states other than Reference_normal--these documents have various errors. * htmerge/docs.cc: If a document has a state of Reference_obsolete, ignore it. * htcommon/HtWordList.h, htcommon/HtWordList.cc (Skip): Change MarkGone() to Skip() to emphasize that this document should be ignored. Wed Jan 19 14:11:51 2000 Loic Dachary * htword/WordList.cc (SkipUselessSequentialWalking): return OK if skipping, NOTOK if not skipping. * htword/WordReference.h: remove useless Clear in WordReference(key, record) constructor. * htword/WordList.h,WordList.cc: Split Walk in three separate functions WalkInit, WalkNext and WalkFinish. Much clearer. Fill the status field of WordSearchDescription to have more information about the error condition. Add found field to WordSearchDescription for WalkNext result. Add cursor_get_flags and searchKeyIsSamePrefix fields to WordSearchDescription as internal state information. * htword/WordList.h,WordList.cc: WalkInit to create and prepare cursor, WalkNext to move to next match WalkNextStep to move to next index entry, be it a match or not WalkFinish to release cursor. * htword/WordList.h: WordSearchDescription::ModifyKey add to jump while walking. * htword/WordList.cc (WalkNext) : it is now legal to step without collection or callback because search contains the last match (found field) and it s therefore not useless. Mon Jan 17 12:15:45 2000 Gilles Detillieux * contrib/htdig-3.2.0.spec: added sample RPM spec file for 3.2 Sat Jan 15 11:53:35 2000 Loic Dachary * htdb/htstat.cc,htdb/htdump.cc: remove useless -S option since the page size is found in the header of the file. * htdb/htstat.cc,htdump.cc,htload.cc: only call WordContext::Initialize if -W flag specified. Fri Jan 14 18:39:12 2000 Marcel Bosc * htword/WordBitCompress.cc: speedup, VlengthCoder::code() finds appropriate coding interval much faster Fri Jan 14 11:30:41 2000 Gilles Detillieux * htdig/Retriver.cc(IsValidURL): Fix problem with valid_extensions, which got lost in the shuffle yesterday. Fri Jan 14 15:56:49 2000 Loic Dachary * htword/WordType.cc,WordRecord.cc,WordKeyInfo.cc (Initialize): change inverted test on instance (== instead of !=). * htword/WordRecord.cc (WordRecordInfo): change inverted test on compare Fri Jan 14 14:24:39 2000 Loic Dachary * htdig/htdig.cc,htmerge/htmerge.cc,htsearch/htsearch.cc: Use Initialize(defaults) to load configuration file if provided. * htword/WordDBCompress.cc (Compress): initialize monitor to null in constructor and check if null before usage. Core dumped in htdb/htload. * htword/WordContext.h (class WordContext): Add Initialize(const ConfigDefaults* config_defaults = 0) that probe configuration files. Usefull when htword is used as a standalone library. Thu Jan 13 19:52:27 2000 Geoff Hutchison * htdig/Retriver.cc: Fix problem with valid_extensions when an "extension" would include part of a directory path or server name, as contributed by Warren Jones. Thu Jan 13 19:22:25 2000 Geoff Hutchison * htnet/Makefile.am, htnet/Makefile.in: Add HtFile to the build process. Thu Jan 13 18:58:03 2000 Geoff Hutchison * htnet/HtFile.h, htnet/HtFile.cc: New Transport classes contributed by Alexis Mikhailov to allow file:// access. * htdig/Document.h, htdig/Document.cc: Add logic to call HtFile objects for URLs. * htcommon/URL.cc: Don't remove a trailing index.html (removeIndex) if the URL is a file://URL. Thu Jan 13 18:49:41 2000 Geoff Hutchison * contrib/conv_doc.pl, contrib/parse_doc.pl: Replace "break" by "last" for correct Perl syntax and additional cleanups and simplifications as contributed by Warren Jones. Thu Jan 13 18:42:29 2000 Geoff Hutchison * htword/WordType.h, htword/WordType.cc: Implementation of new methods IsDigit() and IsCntrl() as contributed by Marc Pohl . Fixes some problems with 8-bit characters. Thu Jan 13 17:17:47 2000 Geoff Hutchison * ChangeLog.0, configure, configure.in, htfuzzy/Endings.cc, htlib/String.cc, htlib/Configuration.cc, htlib/QuotedStringList.cc, htlib/regex.c, htcommon/defaults.cc, htdig/ExternalParser.cc, htdig/Retriever.h, htsearch/Display.cc, include/htconfig.h.in installdir/htdig.conf: Merge in changes from 3.1.x releases. * htdoc/: Merge in documentation changes from 3.1.x releases. Thu Jan 13 20:12:42 2000 Loic Dachary * htword/WordList.cc (Walk): close the cursor before returning. If not doing that the cursor might be closed after the database is closed, leading to double free of the cursor. Bad bug. Thu Jan 13 13:23:17 2000 Loic Dachary * htword/WordContext.h (class WordContext): simplifies a lot. WordContext is no longer a repository for pointers of class instances. Only a place to call Initialize for classes that have a single instance. * htlib/HtWordType.cc: added to include definition of functions shortcuts for WordType. * htword/WordRecord.h,WordType.h,WordKeyInfo.h: implement homogeneous scheme to handle unique instance of the class. - constructor takes const Configuration& argument and init object with config values - static member instance - static method Initialize the static member instance - static method Instance returns the pointer in instance data member * htword/WordRecord.cc: add constructor for WordRecordInfo, and Instance static function. Add WORD_RECORD_INVALID to depict uninitialize WordRecordInfo object. * htword/WordKeyInfo.h: rename SetKeyDescriptionFromFile and SetKeyDescriptionFromString to InitializeFromFile and InitializeFromString and implement them by calling Initialize. rename SetKeyDescriptionRandom to InitializeRandom rename Initialize(String& line) to GetNFields(String& line) rename Initialize(int nfields) to Alloc(int nfields) * htdig/htdig.cc,htmerge/htmerge.cc,htsearch/htsearch.cc,test/word.cc: replace WordList::Initialize with WordContext::Initialize and run immediately after config is read. Otherwise WordType fails to work and configuration value extraction will fail. * htmerge/htmerge.cc: move initialization * test/conf/htdig.conf2.in: reorder so that it looks as much as possible as conf.in Thu Jan 13 12:33:46 2000 Loic Dachary * htdb/htstat.cc,htdump.cc,htload.cc: set proper progname Wed Jan 12 20:02:26 2000 Loic Dachary * htcommon/HtWordList.cc (Dump): Use Walk instead of Collect otherwise does not work. Wed Jan 12 19:38:33 2000 Loic Dachary * htlib/HtDateTime.h (class HtDateTime): killed void SetDateTime(const int t) because they cause problems when time_t is an int and were useless anyway. Wed Jan 12 13:31:45 2000 Loic Dachary * htword/WordBitCompress.h: remove inline qualifier on check_tag1: its not inline * htword/WordKey.h: #define WORD_KEY_UNKNOWN_POSITION to -1. Remove default argument to SetToFollowing so that its more explicit when used with WORD_KEY_UNKNOWN_POSITION. * htword/WordKey.cc: change name of variable info0 to info * htword/WordList.cc: use WordKey::Info instead of WordKeyInfo::Get as done in WordKey.cc for consistency. * htword/WordList.{cc,h},htword/WordDB.h: rename WordCursor to WordDBCursor for consistency. * htword/WordList.h: Kill the WordSearchDescription::Setup useless function * htword/WordList.h: WordSearchDescription constructor now have a straightforward semantics. * htword/WordList.h: Rename Search into Collect since it already existed, just with a different prototype. Wed Jan 12 12:36:46 2000 Loic Dachary * htword/WordList.h (class WordSearchDescription): add cursor member Tue Jan 11 19:33:44 2000 Marcel Bosc * htlib/HtVectorGeneric,htword: Fixed some warnings found when compiling under FreeBSD Tue Jan 11 18:22:58 2000 Marcel Bosc * htlib/HtVectorGeneric.h: inlined functions Add and Allocate which are critical to performance Tue Jan 11 12:18:47 2000 Marcel Bosc * htword/WordKey.h: fixed uninitialized memory read * htword/WordBitCompress.cc: Fixed big number BUG Fixed memeory leak Tue Jan 11 09:37:36 2000 Loic Dachary * htword/WordList.h: move operator << and operator >> to end of functions declarations instead of data members. * htword/WordList.h: added more comments on functions behaviour. * htword/WordList.h: added #if SWIG for Perl interface Mon Jan 10 17:55:05 2000 Marcel Bosc * htword/WordDBPage: enhanced compression debugging output Mon Jan 10 09:07:19 2000 Loic Dachary * WordContext.h,WordKey.h,WordList.h: Added #if SWIG for perl interfaces. Remove InSortOrder, useless now that everything is manipulated in sort order as far as the interface is concerned. * WordKey.cc,WordList.cc: remove InSortOrder * WordKey.h,WordRecord.h,WordReference.h: commented out Set/Get for ascii Set/Get for SWIG. * WordKey.h: turn CopyFrom to public for those who dont want to use operator =. * WordKey.h: rename info -> Info and nfields NFields * WordKey.h: remove int IsFullyDefined() const redundant with Filled Thu Jan 06 14:41:15 2000 Marcel Bosc * htword,all: Changed interface to overloaded Walk function that was ambigous on some compilers... Thu Jan 06 14:00:01 2000 Loic Dachary * htword/WordList.h (class WordSearchDescription): rename setup to Setup * htword/WordList.h (class WordBenchmarking): rename show to Show * htword/WordRecord.{h,cc}, htword/WordReference.h, htword/WordList.h: add comments, reorganize member functions for clarity. Thu Jan 06 12:01:47 2000 Marcel Bosc * htword/compression: Split WordDBCompress.* to WordDBCompress + WordDBPage.* * htword/WordBitCompress: renamed put/get to put_uint/get_uint. added get/put_uint_vl * htword/compression: modified slightly the compression: this makes old databases OBSOLETE: headers compress better. Chaged Flags compress better and faster. * htword/WordKey: added operator [] and Get/Set accessors * htword: removed the obsolete --with_key configure option (KEYDESC) * htword/WordMonitor: addded monitor input Wed Jan 05 14:32:31 2000 Loic Dachary * htword/WordKeyInfo.h (class WordKeyInfo ): if(encode) was if(sort) * htword/WordKeyInfo.h: rename show to Show an nprint to Nprint * htword/WordKeyInfo.h: move WORD_ISA from WordKey.h to WordKeyInfo.h, rename WORD_ISA_String to WORD_ISA_STRING. * htword/WordKey.h: rename FATAL_ABORT to WORD_FATAL_ABORT and errr to word_errr * htword/WordKey.h: move private functions at bottom of class above data members rename show_packed to ShowPacked * htword/WordKey.cc: move WordKeyInfo::SetKeyDescriptionRandom from WordKey.cc to WordKeyInfo.cc * htword/WordKeyInfo.cc: add include htconfig.h Wed Jan 05 13:26:16 2000 Loic Dachary * htdig/ExternalParser.cc (parse): use nocase_compare instead of mystrcasecmp to suppress warnings. (char*)String for mystrncasecmp that has no equivalent in the String class. * htdig/Retriever.cc (IsValidURL): remove warning by (char*)url Wed Jan 05 11:54:19 2000 Loic Dachary * htword/WordKey.h: kill obsolete comment and add suffix explanation at the beginning of the file. * htword/WordKey.h (class WordKey): rename copy_from and initialize to CopyFrom and Initialize to fit naming conventions. Reorganize the methods to group them in logical sets. Fix indenting. Comment each method. * htword/WordKey.h (Clear): add kword.trunc() * htword/WordKey.h: protect SetWord(const char *str,int len) because it opens the door to all kind of specific derivations. Should be SetWord(String(foo, foo_length)) if not performance critical. Wed Dec 29 18:41:14 1999 Marcel Bosc * htlib/HtMaxMin: added max/min of arrays, added comments to HtMaxMin. Added HtMaxMin.cc all these are used in htword * htlib/HtTime.h: added comments. included portable time.h * htlib/HtVectorGeneric.cc: added HtVector_double, HtVector_String * htlib/HtVectorGeneric.h: inlined several methods, disactivated CheckBounds * htlib/StringMatch.cc: removed #include"WordType.h", this made htlib dependant on htword, which is not acceptable for a library * htlib/HtWordType.h: this replaces the macros used in StringMatch.cc * htlib/HtRandom.h: added tools for using random number (this is used currently in tests) * htword/WordBitCompress.cc: transfered max_v/min_v to htlib * htword/WordBitCompress.cc: optimized put/get for better performance * htword/WordMonitor: system for detailed monitoring of operation and performance within htword * htword/WordDBCompress: fixed compression for case of empty WordRecord * htword/WordDBCompress: cleaned up some code added some comments * htword/WordKeyInfo: split WordKey files into WordKey and WordKeyInfo files * htword/WordContext: centralized global configuration into one class * htword/WordKey: inserted randomized key/keydescription into WordKey classes (this was previously used in several tests) * htword/WordKey: optimized Compare, UnpackNumber for speed (these are really speed critical) * htword/WordRecord: is now configurable, type can be configured to "DATA" (htdig) or "NONE" (for other uses) * htword/WordType: changed macros to global functions to make it compatible with cleanup in StringMatch. Integrated WordType to WordContext configuration/Initialization * htword/WordKeyInfo: fixed initialization from key descrition file Tue Dec 28 18:58:21 EET 1999 Vadim Chekan * htlib/String.cc: String::lowercase(), String::uppercase() support for national character added. * htfuzzy/Prefix.cc: method "prefix" works now. Mon Dec 27 22:17:48 1999 Loic Dachary * htdig/htdig.cc (main): change '\r\n' to "\r\n" * Makefile.config,db/dist/Makefile.in: rename libdb to libhtdb to prevent conflicts with installed libdb. * db/dist/Makefile.in: do not install documentation nor binary utilities (db_dump & al) since they are replaced by htdb binaries (htdump & al). * db/dist/Makefile.in (prefix): prepend $(DESTDIR) to prefix to support make DESTDIR=/staging install for binary distribution packages generation. * configure.in: use AC_FUNC_ALLOCA to check for alloca. Used in regex and test/dbbench.cc only but definitely a usefull feature to have. Thu Dec 23 11:10:24 1999 Marcel Bosc * htcommon/defaults.cc: set wordlist_cache_size default to 10Meg * db/mp: removed some debuging messages * htword/WordList.cc: added warning if no cache * test/word.cc: added cache * htlib/HtTime.h: added ifdefs for portable time.h sys/time.h Tue Dec 21 23:33:06 1999 Loic Dachary * htdoc/attrs.html,cf_by*.html: regenerate to include wordlist_wordkey_description attribute * htcommon/Makefile.am: Add AM_LFLAGS = -L and AM_YFLAGS = -l to prevent #line generation because it confuses the dependencies generator of GCC if configure run out of source tree. * configure.in: remove --with-key option. Not needed since word description now dynamic. Destroyed WordKey.h if specified. * htword/Makefile.am: remove commented lines for WordKey.h generation. Tue Dec 21 18:18:01 1999 Marcel Bosc * htword: added code for benchmarking Mon Dec 20 17:59:15 1999 Marcel Bosc * WordKey: Made the key structure dynamic: Changing the key structure used to imply recompiling the htword library. This should not change anything in htdig. * WordKey: numerical key fields are stored in an array of unsigned ints instead of compile-time defined pools. * WordKey.h: WordKey now needs copy opreators. Setbits are stored in sort order (used to be in encoding order) * htword: word_key_info is now a pointer, had to change all references * word.cc: Rewrote wordkey test for new dynamically set key structure. The test randomly creates key structures and tests them. * test: adapted test files (simplifies things a lot) 1999-12-21 Toivo Pedaste * htlib/Dictionary.cc: Fix memory leak when destroying dictionary * htlib/StringList.cc, htdig/Retriever.cc: Fix memory leak, not the most elegent way but I'm not sure about the exact semantics of StringList Mon Dec 20 21:59:03 1999 Loic Dachary * htdb/{Makefile.am,err.c,getlong.c}: Fix mistake: err.c and getlong.c contain C functions (declared in clib_ext) and must be C compiled otherwise the prototype won't fit. Checking db Makefiles, getlong.c and err.c are added to the list of objects for each utility program. This guaranties that they won't conflict with objects included in libdb.a. Sun Dec 19 20:04:42 1999 Loic Dachary * htdb/{Makefile.am, err.cc}: add err.cc for portability purposes. Fri Dec 17 18:04:09 1999 Loic Dachary * Makefile.config: add PROFILING variable and document it. Designed to enable profiling of htdig easily. * */Makefile.am: add *_LDFLAGS = $(PROFILING) for every binary to enable profiling, if specified. Thu Dec 16 17:16:33 1999 Loic Dachary * htdb/*.cc: add -W option to activate htword specific compression. Keep compatibility with zlib compression (-z only). Thu Dec 16 11:56:02 1999 Loic Dachary * test/dbbench.cc: change wrong strcpy with memcpy Wed Dec 15 15:04:39 1999 Gilles Detillieux * htdig/htdig.cc(main): Handle list of URLs given on stdin, if optional "-" argument given. (Uses >> operator below.) * htlib/htString.h, htlib/String.cc: Added Alexis Mikhailov's String input methods, readLine() and >> operator. Wed Dec 15 13:59:34 1999 Gilles Detillieux * htdig/Retriever.cc: remove include of sys/stat.h, which is no longer needed after hack removed from Need2Get(), and could pose a problem on systems that need sys/types.h included first. Wed Dec 15 17:00:04 1999 Loic Dachary * htword/WordDB.h: add inline keyword for portability * htword/WordDB.h: add CmprInfo method to get object describing compression scheme for Berkeley DB * htdb: Add htdump, htload, htstat equivalent of db_dump db_load and db_stat that know about htword specific compression strategy. * htword/WordDBCompress: add static to localy defined functions and variables, remove unecessary #define and #include from header. Tue Dec 14 21:56:57 EET 1999 Vadim Chekan * htcommon/conf_parser.lxx, htcommon/conf_lexer.cxx: bcopy on Solaris is in strings.h, not in string.h. Added check for #ifdef HAVE_STRINGS_H Tue Dec 14 19:18:22 1999 Marcel Bosc * WordBitCompress: code cleaned up and commented Tue Dec 14 18:32:21 1999 Loic Dachary * htword/Word{Record,Reference,Key}: added a Get method to convert the structure into it's ascii string representation. operator << now uses Get. Tue Dec 14 17:46:33 1999 Loic Dachary * db/dist/Makefile.in (install): fix bugous test for libshared Tue Dec 14 14:10:28 1999 Loic Dachary * htword/{WordKey,WordReference,WordRecord}: rework the input methods (operator >>). Each class now has a Set function to initialize itself from an ascii description and a Get function to retrieve an ascii description of the object. * htword/WordList: operator >> has a better and cleaner input loop using StringList and String instead of char*. Tue Dec 14 12:06:24 1999 Marcel Bosc * WordDBCompress.cc : Added compression version checking Mon Dec 13 21:09:31 EET 1999 Vadim Chekan * htcommon/conf_parser.lxx, htcommon/conf_lexer.cxx: Added #include Without it failed to compile on Solaris. Mon Dec 13 16:31:27 1999 Marcel Bosc * htword/WordBitCompress.cc : fixed bug that made compression fail on big documents or big number of url's ... Mon Dec 13 13:49:35 1999 Loic Dachary * htword/WordKey.h.tmpl: Added *_POSITION macro generation Mon Dec 13 11:51:50 1999 Marcel Bosc * htcommon/conf_parser.yxx: fixed several delete that should be delete [] Sun Dec 12 17:14:00 EET 1999 Vadim Chekan * htcommon/conf_lexer.lxx, htcommon/conf_lexer.cxx: national symbols are allowed in right part of expressions (noted by Marcel Bosc). Changed default behavior of flex from print unknown chars on stdout to exit with error message. Sat Dec 11 17:34:03 EET 1999 Vadim Chekan * htdig/Retriever.cc,htdig/htdig.cc: "exclude_urls","bad_querystr" "bad_extensions","valid_extensions","local_default_doc" changed for new config. * htdig/Server.cc: "server_max_docs","server_wait_time" changed for new config. * check for "limit_normalized" moved from Retriever::got_href and Retriever::got_redirect to more appropriate Retriever::IsValidUrl Fri Dec 10 18:05:48 1999 Marcel Bosc * htword: checked for failed memory allocations in compression code Fri Dec 10 18:03:42 1999 Marcel Bosc * htword/WordList,htcommon/HtWordList.cc,htmerge/words.cc: cleaned up WordList::Walk() function, change two occurences of WordList::Walk in htdig files Fri Dec 10 17:40:22 1999 Marcel Bosc * htword/WordKey.cc (Compare): Fixed bug: compare used to compare chars and not unsigned chars, this failed when non-ascii caracters were used Fri Dec 10 11:54:36 1999 Marcel Bosc * htcommon/defaults.cc : doc for wordlist_cache_size Thu Dec 09 17:07:47 1999 Marcel Bosc * htcommon/defaults.cc: added defaults for compression and DB configuration parameters Thu Dec 09 16:47:54 1999 Loic Dachary * db/dist/configure.in,Makefile.in: Added shared lib support for linux only. Not enabled if not on linux. Thu Dec 09 15:07:11 1999 Loic Dachary * acinclude.m4,db/dist/acinclude.mr: CHECK_ZLIB now fails if either zlib.h or libz is not found. * configure.in: do not test zlib.h * db/db/db.c,db/mp/mp_fopen.c: added #ifdef HAVE_ZLIB so that compilation works if zlib is not found * htlib/.cvsignore: remove wrong *.cxx * test/dbbench.cc: added #ifdef HAVE_ZLIB so that compilation works if zlib is not found Thu Dec 09 13:25:45 1999 Marcel Bosc * test/Word.cc,t_wordlist,Makefile.am: upgraded tests * htcommon/HtWordList.h: fixed Configuration/HtConfiguration problem Thu Dec 09 12:10:32 1999 Marcel Bosc * htword: Added the compression code: * WordDBCompress: Classes for page specific compression code * WordBitCompress: Classes for bitstreams and non-specific compression Thu Dec 9 12:09:51 EET 1999 Vadim Chekan * htcommon/HtConfiguration.cc: bug fix: sometimes htConfiguration::Find(url,char*) retuned empty values even if there was something to return. Thu Dec 09 11:15:30 1999 Marcel Bosc * htlib/Configuration.cc (Read): Read is now a virtual function: the old one for Configuration the new one (Vadim's ... with the parser) in HtConfiguration Thu Dec 09 11:01:22 1999 Loic Dachary * acinclude.m4: upgrade AC_PROG_APACHE macro for modules detection. * test/conf/httpd.conf,test/test_functions.in,test/conf/Makefile: use @APACHE_MODULES@ to accomodate various apache modules directory flavors. Tue Dec 07 20:32:34 1999 Marcel Bosc * htdig: Split the Configuration class into Configuration and HtConfiguration. All the HtConfiguration and the configuration parsing (lex..) was woved to htcommon. Configuration was replaced by HtConfiguration as needed Tue Dec 07 16:21:13 1999 Loic Dachary * configure.in: added AM_PROG_LEX and AC_PROG_YACC * htlib/Makefile.am: simply set conf_lexer.lxx and conf_parser.yxx, automake knows how to handle these. The renaming is needed to avoid conflicts in automake generated rules. Mon Dec 6 16:23:39 CST 1999 Gilles Detillieux * htdoc/cf_generate.pl: added a bit of error checking for when it can't fetch the config info, and made it more flexible for what it allows as terminator. * htcommon/defaults.cc: add default and description for authorization attribute, and clean up external_protocols entry for cf_generate.pl. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl * htdig/htdig.cc(main): set authorization parameter before Retriever constuctor is called, as it may initialize a Server. (Should complete fix of PR#490.) Mon Dec 6 21:34:29 EET 1999 Vadim Chekan * htdig/Document.cc htdig/htdig.cc: "authorization" parameter in config is added and is new config compatible. New code has'n got PR#490 bug (don't authentificate robot.txt) Mon Dec 06 11:58:56 1999 Marcel Bosc * HtVectorGeneric.h: generic vectors, stl-free: this was originally a copy of HtVector.h with Object * replaced by GType and some small changes. It has been modified and checked to see if it all works ok. You can build vectors of any type that has an empty constructor. * HtVectorGenericCode.h: generic vectors, stl-free: implementation (modified "copy" of HtVector.cc) * HtVectorGeneric.cc: generic vectors: implementation for common types * HtVector_int.h: generic vectors: declaration for the most common type (and example of howto use) Sat Dec 4 23:49:18 1999 Geoff Hutchison * htfuzzy/Synonym.cc (createDB): Change declaration to match Fuzzy::createDB(config), allowing the method to be called by htfuzzy. * htfuzzy/htfuzzy.cc (main): Add an error message if fuzzy->createDB() comes back with an error. Sat Dec 4 15:38:34 EET 1999 Vadim Chekan * htnet/HtHTTP.cc, htnet/HtHTTP.h, htdig/Document.cc fixed proxy bug. GET command in HtHTTP included only path of url insead full url when use proxy. HtHTTP::UseProxy(int) added. * htdig/Document.cc: make "http_proxy" parameter url-depended for new configuration. Fri Dec 03 14:57:13 1999 Marcel Bosc * BerkelyDB: Compression code: added possibility to use user-defined compression routines (the goal is to enable the mifluz-specific DB page compression that obtains higher compression ratios than generic zlib compression) this envolves the following changes in BerkeleyDB: * BerkelyDB/CompressionEnvironment: Adding a structure db_cmpr_info in db_env that permits db user to specify the external compression routines and other information related to compression * BerkelyDB/CompressionEnvironment: Adding a cmpr_context structure to DB_MPOOLFILE that stores information that compression needs (the _weacmpr DB and the db_cmpr_info) * BerkelyDB/Compression: Needed to modify the compression system (that is implemented in the BerkelyDB memory pool) to permit higher compression ratios and to use the compression environment Thu Dec 2 16:47:30 1999 Geoff Hutchison * htdig/Retriever.cc(parse_url): Use a static int to avoid re-fetching local_urls_only from the config object. (Initial, got_href, got_redirect): Try to get the local filename for a server's robots.txt file and pass it along to the newly generated server. * htdig/Server.cc(ctor): Retrieve the robots.txt file from the filesystem when possible and respect the local_urls_only option. * htdig/Server.h: Change type of local_robots_file to String* to better match Retriever::GetLocal(). Thu Dec 02 16:24:27 1999 Loic Dachary * htword/WordReference.cc,WordKey.cc,WordRecord.cc (Print): Add function to ease printing from Perl. Thu Dec 02 16:06:29 1999 Loic Dachary * htword/WordReference.h (WORD_FILLED): remove unused WORD_FILLED and WORD_PARTIAL macros Wed Dec 01 19:18:42 1999 Loic Dachary * htword/WordKey.h.tmpl,WordRecord.h,WordReference.h, WordList.h: Added #ifndef SWIG for www.swig.org sake. Wed Dec 1 19:47:20 1999 Geoff Hutchison * htlib/HtRegex.cc, htlib/HtRegex.h (set*): Add a case_sensitive flag which defaults to insensitive. This better mirrors the StringMatch class. * htcommon/URL.cc(signature): Make the signature a proper URL to the base of the server. * htdig/Server.h: Add IsDead() methods to query the status of the server, as well as an IsDisallowed() method to query whether a URL is forbidden by the robots.txt rules. Change _disallow to HtRegex. * htdig/Server.cc(ctor): Only retrieve the robots.txt file if this is an http or https server. (robotstxt): Use the proper HtRegex method for setting the pattern. (push): Remove logic checking the _disallow patterns. This is now done by the Retriever object. * htcommon/defaults.cc: Add new attribute "local_urls_only" which defaults to false, which dictates whether retrieval should revert to another method if RetrieveLocal() fails. * htdig/Retriever.cc(parse_url): Check to see if the server is dead before calling the Retrieve() method. Notify the server object if a connection fails. Also respects the new local_urls_only attribute as described above. (IsValidURL): Check the server's IsDisallowed() method to see if the robots.txt forbids this URL. * htdoc/THANKS.html: Updated to reflect current contributions, etc. * README: Update to mention version 3.2.0b1. Wed Dec 1 17:05:48 1999 Gilles Detillieux * htdig/Retriever.cc(GetLocal): Fix error in GetLocalUser() return value check, as suggested by Vadim. Wed Dec 1 15:57:09 1999 Gilles Detillieux * contrib/conv_doc.pl: Added a sample external converter script. Mon Nov 29 23:19:35 1999 Geoff Hutchison * htdig/Retriver.cc, htdig/Retriver.h, htdig/Server.cc, htdig/Server.h: forward-ported patch provided by Alexis Mikhailov and Gilles's for cleaning up IsLocal/GetLocal. Makes local digging persistent, even when HTTP server is down. Mon Nov 29 22:35:06 1999 Geoff Hutchison * test/url.cc: New test for URL class. * test/url.parents: Base URLs for parsing. * test/url.children: Derived relative URLs for testing. * test/Makefile.am, test/Makefile.in: Add the above for building. * htcommon/URL.cc: A variety of bug fixes (some hacks), especially for file:// and user@host URLs. Sun Nov 28 00:35:59 1999 Geoff Hutchison * .version: Bump to 3.2.0b1-dev. Sat Nov 27 20:23:14 1999 Geoff Hutchison * htdig/ExternalTransport.h, htdig/ExternalTransport.cc: New class to allow external scripts to handle transport methods. * contrib/handler.pl: Example handler using the program 'curl' to handle HTTP or HTTPS transactions. * htcommon/defaults.cc: Add new configuration option 'external_protocols' as a list of protocols and scripts to handle them. Documentation currently needs to be written. * htdig/Document.h, htdig/Document.cc(Retrieve): Call ExternalTransport::canHandle to establish which protocols are supported by handler scripts and then create an appropriate transport object. * Makefile.in, htdig/Makefile.am, htdig/Makefile.in: Add dependencies for ExternalTransport class. * htnet/HtHTTP.h, htnet/HtHTTP.cc, htnet/Transport.h, htnet/Transport.cc: Move _location field from HtHTTP_Response to Transport_Response to allow other subclasses to use it. Similarly, move NewDate and RecognizeDateFormat to Transport. Fri Nov 26 17:07:52 1999 Gilles Detillieux * htdig/HTML.cc(HTML & do_tag): add code to turn off indexing between tags. Fri Nov 26 15:56:47 1999 Gilles Detillieux * htsearch/Display.cc(setVariables): added Alexis Mikhailov's fix to check the number of pages against maximum_pages at the right time. * htlib/String.cc(write): added Alexis Mikhailov's fix to bump up pointer after writing a block. Wed Nov 24 15:10:05 1999 Geoff Hutchison * installdir/htdig.conf: Add bad_extensions to make it more obvious to users how to exclude certain document types. Tue Nov 23 19:29:37 CST 1999 Gilles Detillieux * htnotify/htnotify.cc(send_notification): apply Jason Haar's fix to quote the sender name "ht://Dig Notification Service". Tue Nov 23 19:46:00 EET 1999 Vadim Chekan * conf.tab.cc.h conf.l.cc conf.tab.cc Added files pre-generated from conf.y, conf.l Sun Nov 21 18:26:21 EET 1999 Vadim Chekan *htdig/Document.cc: "max_doc_size" supports new configuration and is url-depended now. Sun Nov 21 17:06:50 EET 1999 Vadim Chekan * New config parser commited. htlib/(Makefile.am,Makefile.in), htlib/Configuration.cc, htlib/Configuration.h htlib/(conf.y, conf.l) added. Fri Nov 12 14:17:37 1999 Gilles Detillieux * htcommon/cgi.cc(init): Fix bug in reading long queries via POST method (PR#668). Wed Nov 10 15:34:04 1999 Gilles Detillieux * htsearch/Display.cc(setVariables & createURL), htsearch/htsearch.cc(main), htdoc/hts_templates.html: handle keywords input parameter like others, and make it propagate to followups. Wed Nov 10 15:16:57 1999 Gilles Detillieux * htdig/Retriever.cc: Fix PR#688, where htdig goes into an infinite loop if an entry in local_urls (or local_user_urls) is missing a '=' (or a ','). * htcommon/defaults.cc: removed vestigial references to MAX_MATCHES template variables in search_results_{header,footer}. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl * htdoc/hts_form.html: add disclaimer about keywords parameter not being limited to meta keywords. * htdoc/meta.html: add description of "keywords" meta tag property. add links to keywords_factor & meta_description_factor attributes. 1999-11-10 Toivo Pedaste * htdig/Retriever.cc : Ignore SIGPIPEs with persistant connections * htnet/HtHTTP.cc : Fix buffer overrun reading chunks * htdig/Document.cc : Make redirects work * htdig/Retriever.cc : Make valid URL checks apply to initial URL's particularly those from a previous run * htlib/Dictionary.cc : Fix memory deallocation error Tue Nov 02 13:44:57 1999 Marcel Bosc * htsearch/Display.cc (setVariables): parentheses missing around ternary operator : confusion in priority with <<. Tue Nov 02 13:33:50 1999 Marcel Bosc * htsearch/Display.cc (hilight): changed static char * (!!) to const string, static char evaluated before configuration is loaded so config had no effect + unnecesary conversion Tue Nov 02 11:45:49 1999 Marcel Bosc * htword/WordKey.cc : Cleaned up obsolete code now using *InSortOrder fcts and WordKeyInfo.sort[] * htword/WordKey : Added FirstSkipField : find first field that must be checked for skip * htword/WordKey (PrefixOnly): now returns OK/NOTOK, fixed bug which made Walk loop over the whole db if the searchkey just had a the "word" field defined * htword/WordKey.cc (Unpack): had forgten to: SetDefinedWordSuffix * htword/WordKey.cc (operator >>): added check for very very long words (even if this should never happen) * htword/WordKey.cc (operators << >>): added word suffix handling * htword/WordKey.h : Filled() did not check for WordSuffix * htword/WordKey.h : added WordKey::ExactEqual * htword/WordKey.h (IsDefinedWordSuffix): fixed bad flag check * htword/WordList : Removed all obsolete HTDIG_WORDLIST flags: only two remain : COLLECTOR and WALKER the rest is now specified by the searchKey removed action arg to WordList::Collect() * htcommon/HtWordList.cc,htmerge/words.cc : changed flags in calls to WordList::Walk * htword/WordList.cc : skip now deals with the SuffixUndefined case Fri Oct 29 17:13:21 1999 Gilles Detillieux * htdoc/cf_generate.pl: now updates last modified date in attrs.html * htdoc/attrs.html: reran cf_generate.pl Fri Oct 29 15:28:22 1999 Gilles Detillieux * htsearch/Display.cc(setVariables & hilight): added Sergey's idea for start_highlight, end_highlight & page_number_separator attributes. * htcommon/defaults.cc: added & documented these. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl Thu Oct 28 13:06:23 1999 Gilles Detillieux * htdig/ExternalParser.cc: added support for external converters as extension to external_parsers attribute. * htcommon/defaults.cc: Updated external_parsers with new description and examples of external converters. Thu Oct 28 12:52:28 1999 Gilles Detillieux * htcommon/defaults.cc: Updated programs lists for *_factor, so they all refer to htsearch and not htdig. Added htsearch to programs lists for translate_*. img_alt_factor & url_factor not defined yet because they're still not used in htdig/htsearch. Wed Oct 27 15:53:36 1999 Gilles Detillieux * htcommon/defaults.cc: added descriptions & examples for doc_excerpt, heading_factor, max_descriptions, minimum_speling_length, regex_max_words, use_doc_date, valid_extensions. Added references to these elsewhere in document as appropriate. Removed -pairs option from pdf_parser default (again). Minor changes to noindex_start & end, and changed example for modification_time_is_now. Corrected references to heading_factor_[1-6]. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl Wed Oct 27 13:32:50 1999 Gilles Detillieux * htdoc/cf_generate.pl: changed formatting of output to more closely match format of old attrs.html (to make diff'ing easier), and fixed handling of pdf_parser default to strip quotes. * htcommon/defaults.cc: oops, fixed typo in url_part_aliases example. * htdoc/attrs.html, cf_by{name,prog}.html: reran cf_generate.pl Wed Oct 27 18:24:36 1999 Loic Dachary * htdoc/cf_generate.pl: fixed wrong target for cf_byprog, escape HTML chars <>&'" for default values. Wed Oct 27 10:21:18 1999 Gilles Detillieux * htcommon/defaults.cc: restored 2nd example for url_part_aliases Tue Oct 26 16:28:29 1999 Gilles Detillieux * htcommon/defaults.cc: corrected descriptions for allow_in_form, search_results_header, noindex_start, noindex_end. Also fixed a few small typos & formatting errors here & there in descriptions and examples. Tue Oct 26 16:01:22 1999 Loic Dachary * htword/Makefile.am: rm Wordkey.h instead of chmod to copy with non existent WordKey.h Tue Oct 26 10:54:52 1999 Loic Dachary * htcommon/default.cc: fixed all inconsistencies reported by Gilles. Mon Oct 25 11:42:13 1999 Marcel Bosc * htword/ word.cc,t_wordskip,skip_db.txt: Added test for *Skip Speedup* * htword/ WordList: Added tracing of Walk() for debuging purposes Fri Oct 22 18:22:00 1999 Marcel Bosc * htword/ WordList.cc,WordKey: Added a defined/undefined flag for saying if a search key's word is a prefix or not: WORD_KEY_WORDSUFFIX_DEFINED reduces code size and makes it much easier to undertand * htword/ WordList,WordReference,WordKey: Added input output streams for WordList,WordReference,WordKey Wed Oct 20 16:47:52 1999 Marcel Bosc * htword/ WordKey,Makefile.am,WordCaseIsAStatements.h: for readability replaced the switch ... #ifdef ..STATEMENT().... sequence that apeared many times with an include file :WordCaseIsAStatements.h * htword/ WordKey: WordKeyInfo: duplicated all of the fields structure into sort structure, for fast acces without cross referencing and for simplifying code (required change of perl in template WordKey.h.tmpl) * htword/ WordList: *Skip Speedup* added a speedup to avoid wasting time by sequentialy walking through useless entries. see function: SkipUselessSequentialWalking() for an example and more info * htword/ WordKey.h,WordKey.cc: Changed Set,Unset,IsSet Wordkey accesors' names to: SetDefined,Undefined,IsDefined. (easier to read and avoids naming conflicts) * htword/ WordKey: added generic numerical accesors for accesing numerical fields in WordKey (in sorted order):GetInSortOrder,SetInSortOrder * htword/ WordKey,word_builder.pl: added a MAX_NFIELDS constant, that specifies a maximum number of fields that a WordKey can have. Sanity check in word_builder.pl. * htword/ word_builder.pl: enforced word sort order to ascending * htword/ WordList: added a verbose flag using config."wordlist_verbose" Tue Oct 19 18:36:42 1999 Loic Dachary * htword/WordType.h: const accessors to wtype and config Tue Oct 19 13:10:47 1999 Loic Dachary * acconfig.h: remove uncessary VERSION (redundant) Tue Oct 19 11:32:38 1999 Loic Dachary * db/Makefile.in,db/dist/Makefile.in: install db library so that external applications can be linked. Tue Oct 19 10:57:27 1999 Loic Dachary * configure.in: add --with-key to specify alternate to htword/word.desc * configure.in: htword is done before htcommon to prevent unecessary recompilation because WordKey.h changes. * htword/Makefile.am: use @KEYDESC@ Tue Oct 19 10:38:41 1999 Loic Dachary * test/word.cc use TypeA instead of DocID and the like Mon Oct 18 17:21:34 1999 Loic Dachary * Makefile.config: AUTOMAKE_OPTIONS = foreign Mon Oct 18 11:40:17 1999 Marcel Bosc * htword/ WordList.cc (Walk): fixed bug in Walk: if flag HTDIG_WORDLIST was set then data was uninitialized in loop Fri Oct 15 18:52:03 1999 Marcel Bosc * htdig/Document.h (class Document): added const to: Transport::DocStatus RetrieveLocal(HtDateTime date, const String filename); Fri Oct 15 17:46:23 1999 Loic Dachary * acinclude.m4,configure.in: modified AC_APACHE_PROG to detect version number and control it. * test/conf/*.in: patch to fit module loading or not, accomodate various installation configurations. * test/test_functions.in: More portable call to apache. Fri Oct 15 12:55:47 1999 CEST Gabriele Bartolini * htdig/Document: added the management of 'persistent_connections', 'head_before_get', 'max_retries' configuration attributes. Fri Oct 15 12:54:11 1999 CEST Gabriele Bartolini * test/testnet.cc: added the option '-m' for setting the max size of the document. Fri Oct 15 12:48:49 1999 CEST Gabriele Bartolini * htdig/Server: added a flag for persistent connections. It's set to true if the Server allows persistent connections. It should be used when retrieving a document. Fri Oct 15 12:45:42 1999 CEST Gabriele Bartolini * defaults.cc: added the configuration attributes 'persistent_connections', 'max_retries' and 'head_before_get'. Their default values are respectively true, 3, false. Fri Oct 15 12:35:51 1999 CEST Gabriele Bartolini * HtHTTP.cc: managing of uncompleted stream reading with persistent connections (it occurs when max_doc_size is lower than the real content length of the document, or when a document is not parsable and we asked for it with a GET call). * Transport: _host variable is treated as a String, as Loic suggested. Fri Oct 15 12:11:23 1999 Marcel Bosc * Added README to htword Thu Oct 14 11:29:35 1999 Geoff Hutchison * htlib/mktime.c, htlib/regex.c, htlib/regex.h, htlib/strptime.c: Updated with latest glibc versions. Merging from glibc sources may have introduced bugs, so this is the last merge before htdig-3.2.0b1. Thu Oct 14 13:09:32 1999 CEST Gabriele Bartolini * htnet/Transport: added statistics for open and close of connections and changes of servers. Fixed a bug in the SetConnection method, regarding the host comparison. Added a method for showing the statistics on a given channel. * htnet/HtHTTP: More debug info available. Added a method for showing the statistics on a given channel. * test/testnet.cc: now receives changes above. Wed Oct 13 13:35:42 1999 CEST Gabriele Bartolini * htdig/Document.h: added an HtHTTP pointer to the class. * htdig/Document.cc: Transport and HtHTTP initialization methods inside the Document constructur. The class destructor now calls only the HtHTTP destructor (not the Transport destructor). Modified the Retrieve method. * htdig/Server.h: _last_connection is now an HtDateTime object. * htdig/Server.cc: _modified the constructor and the delay method. * htdig/Retriever.cc: modified the parse_url function in order to manage all the Document status messages coming from the Transport class. Also modified the method for not found URLs for managing the no_port status. Tue Oct 12 10:12:10 1999 Loic Dachary * install headers and libraries so that htdig libraries may be used by external programs * htword/WordList.cc,WordType.cc: add comments about config parameters used. Fri Oct 8 09:35:30 1999 Geoff Hutchison * htlib/HtDateTime.cc (SetFTime): Change buffer argument to const char* to prevent problems passing in const buffers. * htnet/HtHTTP.h: Change SetUserAgent to take a const char* to prevent problems passing in const parameters. * htdig/Document.h, htdig/Document.cc(): Use Transport class for obtaining documents. Remove duplication of declarations (e.g. DocStatus). * htdig/Retriever.cc: Adapt switch statements from Document::DocStatus to Transport::DocStatus. * htdig/Server.cc: Use Document::Retrieve instead of RetrieveHTTP. Fri Oct 08 16:35:16 1999 Loic Dachary * test/t_htnet: succeed if timeout occurs. It was the opposite. * configure.in: AC_MSG_CHECKING(how to call getpeername?) add missing comma at end for header spec block. Fri Oct 08 14:42:47 1999 Loic Dachary * Fix all warnings reported by gcc-2.95.1 related to string cast to char*. Fri Oct 08 14:04:21 1999 Loic Dachary * htlib/Configuration,ParsedString,Dictionary: change char* to String where possible. * Fix a lot of warnings reported by gcc-2.95.1 related to string cast to char*. * Completely disable exception code from db. Fri Oct 08 13:44:32 1999 CEST Gabriele Bartolini * HtHTTP.cc: fixed a little bug in setting the modification time if not returned by the server. Fri Oct 08 11:30:53 1999 CEST Gabriele Bartolini * HtHTTP.cc: better management of connection failures return values. * Transport.h: added Document_no_connection and Document_connection_no_port enum values. * testnet.cc: management of above changes. Fri Oct 08 11:27:31 1999 CEST Gabriele Bartolini * configure.in: modified getpeername() test. Fri Oct 08 10:28:15 1999 Loic Dachary * htdig/Retriever.cc (IsValidURL): test return value of ext = strrchr(url, '.'); * htword/WordRecord.h: initialize info member to 0 in constructor and Clear. * htlib/Configuration: char* -> String to all functions. Resolve warnings. Thu Oct 07 16:19:46 1999 Loic Dachary * htnet/HtHTTP.cc (ReadChunkedBody): use append instead of << because buffer is *not* null terminated. * htnet/Transport.cc (Transport): initialize _port and _max_document_size otherwise comparison with undefined value occurs. Thu Oct 07 16:34:21 1999 CEST Gabriele Bartolini * HtHTTP.cc: call FinishRequest everytime in HTTPRequest() a value is returned. * testnet.cc: improved with more statistics and connections timeouts control. Thu Oct 07 12:53:12 1999 CEST Gabriele Bartolini * configure.in: modified getpeername() test function with AC_LANG_CPLUSPLUS instead of AC_LANG_C. Thu Oct 07 11:56:52 1999 CEST Gabriele Bartolini * HtHTTP.cc : fixed bug of double deleting _access_time and _modification_time objects in ~HtHTTP(). Thu Oct 07 10:17:22 1999 Loic Dachary * htword/WordRecord.h: change (const char*) cast to (char*) * htword/WordKey.h.tmp: fix constness of accessors, const accessor returns const ref. Prevents unecessary copies. Wed Oct 6 23:31:50 1999 Geoff Hutchison * htnet/Connection.h, htnet/Connection.cc: Merge in io class. Connection class was the only subclass of io. * Makefile.in, htlib/Makefile.am, htlib/Makefile.in: Update for removed io class. * htdig/ExternalParser.cc: Add more verbose flags for errors. Wed Oct 06 14:56:34 1999 Loic Dachary * htnet/Connection.cc (assign_server): use free, not delete on strdup allocated memory. * htcommon/URL.cc (URL): set _port to 0 in constructors. Wed Oct 06 12:08:38 1999 Loic Dachary * Move htlib/HtSGMLCodec.* to htcommon to prevent crossed interdependencies between htlib and htcommon Wed Oct 06 12:07:32 1999 Gabriele Bartolini * HtHTTP.cc: patch from Michal Hirohama regarding the SetBodyReadingController() method Wed Oct 06 11:49:15 1999 Loic Dachary * Move htlib/HtZlibCodec.* htlib/cgi.* to htcommon to prevent crossed interdependencies between htlib and htcommon Wed Oct 06 11:40:48 1999 Gabriele Bartolini * HtHTTP: stores the server info correctly and removed some debug info in chunk managing Wed Oct 06 11:39:12 1999 Loic Dachary * Move htlib/*URL* to htcommon Wed Oct 06 10:09:19 1999 Loic Dachary * README: add htword * test/t_htnet: fix variable set problem & return code problem Wed Oct 06 08:53:52 1999 Gabriele Bartolini * Written t_htnet test Tue Oct 5 12:24:43 1999 Geoff Hutchison * db/*: Import of Sleepycat's Berkeley DB 2.7.7. * db/db/db.c, db/include/db.h, db/include/db_cxx.h, db/mp/mp_bh.c: Resolve conflicts created in merge. Tue Oct 05 18:53:13 1999 Loic Dachary * htdig/Display.cc, htword/*.cc: add inclusion of htconfig.h Tue Oct 05 14:54:17 1999 Loic Dachary * htlib/htString.h (class String): add set(char*) * htword/WordKey.cc: define typedefs for key components. Leads to more regular code and no dependency on a predefined set of known types. All types must still be castable to unsigned int. Assume Word of type String always exists. Generic Get/Set/Unset methods made simpler. Added const and ref for Get in both forms. * htword/WordList.cc: enable word reference counting only if wordlist_extend configuration parameter is set. This parameter is hidden because no code uses per word statistics at present. It is only activated in the test directory. * htword/word_list.pl: add mapping to symbolic type names, force and check to have exactly one String field named Word. Mon Oct 04 20:05:35 1999 Loic Dachary * test: add thingies to make test work when doing ./configure outside the source directory. * htword/WordList: Add Ref and Unref to update statistics. Fix walking to start from the end of statistics. All statistics words start with \001, therefore at the beginning of the file and all clustered together. * htword/WordStat: derived from WordReference to implement uniq word statistics. * test/word.cc: test statistics updating. * htword/WordKey.cc: fix bugous compare (returned length diff if key of different length). Mon Oct 04 18:43:56 1999 Gabriele Bartolini * test/testnet.cc: added the option for HEAD before GET control Mon Oct 04 17:33:24 1999 Gabriele Bartolini * htnet/Transport.h .cc: added the FlushConnection() method * htnet/HtHTTP.h .cc: now the Request() method can make a HEAD request precede a GET request. This is made by default, and can be changed by using the methods Enable/DisableHeadBeforeGet(). A configuration option can be raised to manage it. Mon Oct 04 12:43:41 1999 Gabriele Bartolini * htlib/io.h .cc: added a flush() method. * htnet/HtHTTP.cc: manage the chunk correctly, by calling the flush() method after reading it. Mon Oct 04 12:02:24 1999 Loic Dachary * htlib/htString.h: move null outside inline operator [] functions. Fri Oct 01 14:55:56 1999 Loic Dachary * htword/WordRecord: mutable, can also contain uniq word statistics. * htword/WordReference: remove all dependencies related to the actual structure of the key. * htcommon/HtWordReference: derived from WordReference, explicit accessors. * htcommon/HtWordList: derived from WordList, only handles the word cache (Flush, MarkGone). * htdig/HTML.cc (do_tag): add wordindex to have location set in tags * htcommon/DocumentRef.cc (AddDescription): add Location calculation * htword/WordList.cc: add dberror to map Berkeley DB error codes * htsearch/Display.cc (display): initialize good_sort to get rid of strange warning. Fri Oct 01 09:02:11 1999 Loic Dachary * Makefile.config: duplicate library lines to resolve interdependencies. Thu Sep 30 17:56:55 1999 Loic Dachary * htmerge/words.cc (delete_word): Upgrade to use WordCursor. * htword/WordList: Walk now uses a local WordCursor. Many concurent Walk can happen at the same time. * htword/WordList: Walk callback now take the current WordCursor. Added a Delete method that takes the WordCursor. Allows to delete the current record while walking. * db/include/db_cxx.h (DB_ENV): add int return type to operator = * db/dist/configure.in (CXXFLAGS): disable adding obsolete g++ option. * configure.in: enable C++ support when configuring Berkeley DB * htword: create. move Word* from htcommon. move HtWordType from htlib and rename WordType. * htword/WordList: use db_cxx interface instead of Database. Less interface overhead. Get access to full capabilities of Berkeley DB. Much more error checking done. Create WordCursor private class to use String instead of Dbt. Wed Sep 29 20:03:31 1999 Loic Dachary * htlib/lib.h: AIX xlC does is confused by overloaded mystrcasestr that only differ in constness. Only keep const form and use cast where approriate. *sigh* * htlib/htString.h: accomodate new form of Object::compare and Copy. Explicitly convert compare arg to String&, prevent hiding and therefore missing the underlying compare function. * htlib/HtVector.cc (Copy): make it const * htlib/HtHeap.cc: accomodate new form of Object::compare * htcommon/List.h,cc: Add ListCursor to allow many pointers that walk the list to exist in the same program. * htlib/Object.h (class Object): kill unused Serialize + Deserialize. Change unused Copy to const and bark on stderr if called because it is clearly not was is wanted. If Copy is called and the derived class does not implement Copy we are in trouble. Alternatives are to make it pure virtual but it will break things all over the code or to abort but this will be considered to violent. Change compare to take a const reference and be a const. Wed Sep 29 16:51:58 1999 Loic Dachary * acinclude.m4,configure.in,Makefile.config: remove -Wall from Makefile.conf, add the AC_COMPILE_WARNINGS macro in acinclude.m4 and use it in configure.in. * htdoc/default_check.pl: remove, unused Wed Sep 29 13:07:58 1999 Gabriele Bartolini * htnet/Transport: fixed some bugs on construction and destruction * htnet/HtHTTP: the most important add is the decoding of chunked encoded responses, as reported on RFC2616 (HTTP/1.1). It needs to be developed, because it timeouts at the end of the request. Added a function pointer in order to dynamically handle the function that reads the body of a response (for now, normal and chunked, but other encoding ways exist, so ...). Fixed some bugs on construction and added some features like Server and Transfer-encoding headers. Wed Sep 29 13:54:59 1999 Loic Dachary * fix all inline method declarations so that they are always declared inline in the class declaration if an inline definition follows. * acinclude.m4: also search apache in /usr/local/apache/bin by default. * fix various warnings of gcc-2.95, now compiles ok without warnings and with -Wall. * htlib/htString.h: removed commented out inline get * test/testnet.cc: add includes for optarg Tue Sep 28 18:56:36 1999 Loic Dachary * Makefile.config (HTLIBS): libhtnet at the beginning of the list. It matters on Solaris-2.6 for instance. * test/testnet.cc: change times to timesvar to avoid conflict with function (was warning only on Solaris-2.6). * htdig,htsearch,htmerge,test/word are purify clean when running make check. Tue Sep 28 18:23:49 1999 Loic Dachary * htmerge/words.cc (mergeWords): use WordList::Walk to avoid loading ALL the words into memory. * htlib/DB2_db.cc (Open): we don't want duplicates. Big mistake. If DUP is on, every put for update will insert a new entry. * htcommon/WordList.cc (Delete): separate Delete (straight Delete and WalkDelete) to avoid accessing dbf from outside WordList. * htcommon/WordList.cc (Walk): now promoted to public. Tue Sep 28 16:34:56 1999 Loic Dachary * test/word.cc (dolist): Add regression tests for Delete. * htcommon/WordList.cc (Delete): Reimplement from scratch. Use Walk to find records to delete. This allows to say delete all occurence of this word, delete all words in this document (slow), delete all occurences of this word in this document etc. * htcommon/WordList.cc (Walk): extend so that it handles walk for partially specified keys, remains fully backward compatible. It allows to extract all the words in a specific document (slow) or all occurences of a word in a specific document etc. Tue Sep 28 12:56:12 1999 Loic Dachary * htcommon/DocumentDB.cc (Open): report errors on stderr * htmerge/docs.cc (convertDocs): rely on error reporting from DocumentDB instead of implementing a custom one. Tue Sep 28 11:36:28 1999 Gabriele Bartolini * htnet/Transport.h: added the status code and the reason phrase * htnet/HtHTTP.cc .h: removed the attributes above. Read the body of a response if the code is 2xx. Issues the GetLocation() method. Tue Sep 28 10:32:47 1999 Loic Dachary * test/htdocs/set3: create and populate with cgi scripts have bad behaviour (time out and, slow connection). Tue Sep 28 10:20:23 1999 Loic Dachary * test/htdocs: move html files in set1/set2 subdirectories to allows tests that use different set of files. Change htdig.conf accordingly. Tue Sep 28 09:31:12 1999 Loic Dachary * test/Makefile.am: comment test options, add LONG_TEST='y' for lengthy tests, by default run quick tests. * installdir/bad_words: removed it an of : since the minimum word length is by default 3, these words are ignored anyway. Mon Sep 27 20:37:38 1999 Loic Dachary * htlib/HtWordType.h,cc: concentrate knowledge about word definition in this class. Rename the class WordType (think WordReference etc...). Change Initialize to use an external default object. A WordType object may be allocated on its own. Drag functionalities from BadWordFile, Replace and IsValid of WordList, and concentrate them in the WordType::Normalize function. * htcommon/WordList: use the new WordList semantic. WordType is now a member of WordList, opening the possibility to have many WordList object with different configurations within the same program since the constructor takes * htsearch/htsearch.cc (setupWords): Use HtNormalize to find out if word should be ignored in query. Formerly using IsValid. * htlib/String.cc (operator []): fix big mistake, operator [] was indeed last() ! * htlib/String.cc(uppercase, lowercase): return the number of converted chars. * htlib/String.cc(remove): return the number of chars removed. Mon Sep 27 17:43:23 1999 Gabriele Bartolini * Created testnet.cc under test dir for trying the htnet library It's a simple program that retrieves an URL. * htnet/HtHTTP.cc, .h: added a 'int (*) (char *)' function pointer. This attribute is static and it is used under the isParsable method in order to determine if a document is parsable. It must be set outside this class by using the SetParsingController static method. The classic use is to set it to 'ExternalParser::canParse' . Mon Sep 27 10:52:51 1999 Loic Dachary * htmerge/db.cc (mergeDB): delete words instead of words->Destroy() because the words object itself was not freed. Mon Sep 27 10:38:37 1999 Gabriele Bartolini * Created 'htnet' library Mon Sep 27 12:39:24 1999 Loic Dachary * test/word.cc (dolist): don't deal with upper case at present and prevent warning. Mon Sep 27 10:38:37 1999 Gabriele Bartolini * htlib/String.cc: removed compiler warnings * htdig/HtHTTP.h: corrected cvs Id property Mon Sep 27 10:29:58 1999 Loic Dachary * htlib/String.cc (String): make sure *all* constructors set the Data member to 0. * htsearch/parser.cc (score): add missing dm->id = wr->DocID(); strange it did not make search fail horribly. Mon Sep 27 09:46:34 1999 Loic Dachary * test/conf/htdig.conf.in (common_dir): add common_dir so that templates are found in compile directory. * htsearch/parser.cc (phrase): free wordList at end and only allocate if needed. Fri Sep 24 16:35:47 1999 Loic Dachary * htcommon/DocumentDB.ccf (Open): change mode to 666 instead of 664, it's the bizness of umask to remove permission bits. * htlib/URL.cc (removeIndex): Memory leak. do not use l.Release since standard Destroy called by destructor is ok. * htdig/htdig.cc (main): Memory leak. Use l.Destroy instead of l.Release. * htlib/StringList.cc (Join): Memory leak (new String str + return *str). Also change to const fct. * htlib/List.cc (Nth): add const version to help StringList::Join save memory. * htdig/HTML.cc (parse): delete [] text (was missing []) * htlib/HtVector.cc: Most of the boundary tests with element_count (but not all of them) were wrong (> instead of >= for instance). * htlib/HtVector.cc (Previous): limit test cut and pasted from Next and obviously completely wrong. Fix. * htlib/HtVector.cc (Remove): use RemoveFrom, avoid code duplication. * htcommon/DocumentRef.cc (Clear): set all numerical fields to 0, and truncate strings to 0. Some were missing. * htlib/Connection.cc (Connection): free(server_name) because allocated by strdup not new. Fri Sep 24 14:30:21 1999 Loic Dachary * */.cvsignore: update to include .pure, *.la, *.lo, .purify * htlib/String.cc (String): add Data = 0 * htlib/htString.h (class String): add Data = 0 * htlib/String.cc (String): init set to MinimumAllocationSize at least prevents leaking if init = 0. * htlib/String.cc (nocase_compare): use get() instead of direct pointer to Data so that the trailing null will be added. * htlib/Dictionary.cc (DictionaryEntry): free(key) instead of delete [] key because obtained with strdup. * htlib/DB2_db.cc (Close): free(dbenv) because db_appexit does not free this although it free everything else. Thu Sep 23 18:18:40 1999 Loic Dachary * configure.in: add PERL detection & use in Makefile.am Thu Sep 23 14:29:29 1999 Loic Dachary * configure.in: removed unused alloca.h * htcommon/DocumentDB.cc: test isopen in Close instead of before calling Close. Add some const in functions arguments. (Read): change char* args to const String&, changed tests for null pointers to empty(). (Add): Delete the temp class member, use function local temp. (operator []): change char* args to const String& (CreateSearchDB): change char* args to const String& * htcommon/DocumentRef.cc:(AddDescription): Add some const in functions arguments. Use a WordReference as insertion context instead of merely the docid: it contains the insertion context. (AddAnchor): Add some const in functions arguments. * htcommon/DocumentRef.h: Add some const in inline functions arguments. * htcommon/Makefile.am: add WordKey + WordKey.h generation * htcommon/word_builder.pl, word.desc, WordKey.h.tmpl: generate WordKey.h from WordKey.h.tmpl and word.desc * htcommon/WordList.cc: In general remove code that belongs to WordReference rather than WordList and cleanup const + String. (WordList) the constructor takes a Configuration object in argument. (Word -> Replace): Word method replaced by Replace method because more explicit. Now taks a WordReference in argument instead of the list of fields values. (valid_word deleted, IsValid only): Add some const in functions arguments. (BadWordFile): change char* args to const String& (Open + Read -> Open): Open and Read merge into Open with mode argument. change char* args to const String&. (Add): use WordReference::Pack and simply do Put. (operator[], Prefix ...) now take WordReference instead of Word. Autmatic Conversion from Word for compatibility thru WordReference(const Word& w). (Dump): change char* args to const String& (Walk): use WordReference member functions instead of hard coded packing * htcommon/WordRecord.h: move flag definitions to WordReference.h only keep anchor, the reste moved to key. * htdig/Document.cc: change all config[""] manipulations from char* to String or const String (setUsernamePassword): Add some const in functions arguments. * htdig/HTML.cc: change all config[""] manipulations from char* to String or const String. Change null pointer tests to empty(). (transSGML): change char* args to const String& * htdig/HtHTTP.cc: Add error messages for default cases in every switch. * htdig/PDF.cc: (parse) change char* to const String& for config[""] * htdig/Plaintext.cc: (parse) remove unused variable * htdig/Retriever.cc: use WordReference word_context instead of simple docid to hold the insertion context. (Retriever) pass config to WordList initializer. (setUsernamePassword): Add some const in functions arguments. (Initial): change char* args to const String& (parse_url): use WordReference word_context, add debug information. (RetrievedDocument): set anchor in word_context. (got_word): use Replace instead of Word (got_*): Add some const in functions arguments. * htdig/htdig.cc: change all config[""] manipulations from char* to String * htdoc/cf_generate.pl: compute attrs.html, cf_byprog.html and cf_byname.html from ../htlib/default.cc and attrs_head.html attrs_tail.html cf_byname_head.html cf_byname_tail.html cf_byprog_head.html cf_byprog_tail.html Add rules in Makefile.am * htfuzzy: In every programs I changed the constructor to take a Configuration agrument. The openIndex and writeDB had this argument sometime used it, sometimes used the global config. Having it in the contructor is cleaner and safer, there is no more reference to the global config. I also changed some char* to String and const. Most of the program look the same, I won't go into details here :-} * htlib/Configuration.cc: changed separators from String* to String. Simpler. (~Configuration): removed because not needed. (Add): change to String, remove new String + delete for local var. (Find, operator[]): make it const fct, add some const in functions arguments. (Value + Double): killed, replaced by as_integer + as_double from String (Boolean): use String methods + string objects (Defaults): Add some const in functions arguments. * htlib/Configuration.h: add char *type; // Type of the value (string, integer, boolean) char *programs; // White separated list of programs/modules using this attribute char *example; // Example usage of the attribute (HTML) char *description; // Long description of the attribute (HTML) to the ConfigDefaults type. * htlib/Connection.cc: (assign_server) change char* args to const String& * htlib/DB2_db.cc: Merge with DB2_hash. Add compare and prefix functions pointers. Merge OpenRead & OpenReadWrite into Open, keep for compatibility. skey and data are now strings instead of DBT. Remove Get_Next_Seq. Get_Next now returns key and value in arguments. Remove all other Get_Next interfaces. * htlib/Database.h: Compatibility functions for Get_Next Put, Get, Exists, Delete take String args and are inline Add SetPrefix and SetCompare * htlib/Dictionary.cc: Add copy constructor. Add DictionaryCursor that holds the traversal context. Use DictionaryCursor object for traversal without explicit cursor specified. Add constness where meaningfull. * htlib/HtPack.cc: (htPack) format is const, change strtol call to use temporary variable to cope with constness. (htUnpack) dataref argument is not a reference anymore. Not used anywhere and kind of hidden argument nobody wants. * htlib/HtRegex.cc: set, match, HtRegex have const args. * htlib/HtWordCodec.cc: (code) orig is const * htlib/HtWordType.cc,h: statics is made of String instead of char*. Remove static String punct_and_extra from Initialize. * htlib/HtZlibCodec.cc: len is unsigned int * htlib/ParsedString.cc: add constness to function args (get) use String instead of char * htlib/QuotedStringList.cc: inline functions argument variations and add constness. * htlib/String.cc: add constness whereever possible. * htlib/htString.h: Add const get, char* cast, operator []. Add as_double conversion. * htlib/StringList.cc: inline functions argument variations and add constness. * htlib/StringMatch.cc: add constness to function args. * htlib/URL.cc: add constness to function args. (URL): fct arg was used as temp. Change, clearer. * htlib/lib.h: add const declaration of string manipulation functions. Two forms for mystrcasestsr: const and not const. * htlib/strcasecmp.cc: add constness to function args. * htlib/timegm.c: add declaration for __mktime_internal * htmerge/db.cc: change *doc* vars from char* to const String, use new WordList + WordReference interface. * htmerge/docs.cc: change *doc* vars from char* to const String. * htmerge/words.cc: use new WordList + WordReference interface. * htsearch/Display.cc: use empty method on String where appropriate. use String instead of char* where config[""] used. (includeURL): change char* args to const String& * htsearch/ResultMatch.cc: (setTitle, setSortType) change char* args to const String& * htsearch/Template.cc: (createFromFile) change char* args to const String& * htsearch/Template.h: accessors return const String& or take const char* * htsearch/TemplateList.cc: (get) use const String for internalNames. * htsearch/htsearch.cc: use String instead of char* where config[""] used. * htsearch/parser.cc: Initialize WordList member with config global. (perform_push): free the result list after calling score. (score, phrase): use new WordList + WordReference interface. Thu Sep 23 14:29:29 1999 Loic Dachary * htcommon/WordKey.h.tmpl, WordKey.cc: new, describe the key of the word database. * htcommon/word.desc: new, abstract description of the key structure of the word database. * htcommon/word_builder.pl: new, generate WordKey.h from WordKey.h.tmpl * htcommon/WordReference.cc: move key manipulation to WordKey.cc Add Unpack/Pack functions. Add accessors for fields and move fields to private. Add constness where possible. Mon Sep 20 14:50:47 1999 Loic Dachary * Everywhere config["string"] is used, check that it's *not* converted to char* for later use. Keep String object so that there is no chance to use a char* that has been deallocated. Using a String as return for config["string"] is also *much* safer for the great number of calls that did not check for a possible 0 pointer return. * htfuzzy/*.{cc,h}: const Configuration& config member. Constructor sets it. Remove config argument from openIndex & writeDB. The idea (as it was initialy, I guess) is to be able to have a standalone fuzzy library using a specify configuration file. It is now possible and consistent. * htlib/htString.cc: more constness where appropriate. Changed compare to have const String& arg instead of const Object* because useless and potential source of bugous code. * htfuzzy/Regex.cc (getWords): fix bugous setting of extra_word_chars configuration value. It is set to change the behaviour of HtStripPunctuation but this function get the extra_word_chars from a static array initialized at program start by static void Initialize(Configuration & config). Use straight s.remove() instead. Besides, the string was anchored by prepending a ^ that was removed because part of the reserved chars. Mon Sep 20 11:47:05 1999 Loic Dachary * htlib/Configuration.cc (operator []): changed return type to String to solve memory leak. When char* the string was malloced from ParsedString after substitution and never freed. In fact it was even worse : it was free before use in some cases. Sun Sep 19 19:12:44 1999 Loic Dachary * htdoc/cf_generate.pl, htcommon/defaults.cc, htlib/Configuration.h: Change the structure of the configuration defaults. Move description, examples, types, used_by information from attrs.html. Write cf_generate.pl to build attrs.html, cf_byname, cf_byprog from defaults.cc. Makes it easier to maintain an up to date description of existing attributes. About 10 attributes existed in defaults.cc and were not describted in the HTML pages. Add rules in htdoc/Makefile.am to generate the pages if a source changes. Fri Sep 17 19:34:48 1999 Loic Dachary * Makefile.config: add -Wall to all compilation and fix all resulting warnings. * htlib/Connection.cc (assign_server): remove redundant test and cast litteral value to unsigned * htlib/String.cc: add const qualifier where possible. Helps dealing with const objects at an upper level. Fri Sep 17 18:27:57 1999 Alexander Bergolth A few changes so that it compiles with xlC on AIX: * configure.in, include/htconfig.h.in: Add check for sys/select.h. Add "long unsigned int" to the possible getpeername_length types. * htdig/htdig.cc: Moved variable declaration out of case block. * htlib/Connection.cc: Include sys/select.h. * htcommon/WordList.cc: just a type cast * htlib/regex.c: define true and false only if they aren't already * htdig/Transport.{h,cc}: removed inline keywords (inline functions have to be defined and declared simultaneously) * htlib/{mktime.c,regex.h,strptime.c,timegm.c}: change // comments to /* ... */ Tue Sep 14 01:15:48 1999 Geoff Hutchison * htmerge/db.cc: Rewrite to use the WordList functions to merge the two word databases. Also make sure to load the document excerpt when adding in DocumentRefs. * htmerge/docs.cc: Fix bug where ids were not added to the discard list correctly. * htmerge/words.cc: Fix bug where ids were not checked for existance in the discard list correctly. Sun Sep 12 12:27:16 1999 Geoff Hutchison * htcommon/defaults.cc: Remove word_list since that file is no longer used. * htdig/htdig.cc: Ensure -a and -i are followed for the word_db file. Fixes PR #638. Sat Sep 11 00:11:28 1999 Geoff Hutchison * htlib/StringMatch.h: Add back mistakenly deleted #ifndef/#define. Fri Sep 10 23:07:43 1999 Geoff Hutchison * htmerge/*, htcommon/*, htdig/*, htlib/*: Add copyright information. Fri Sep 10 11:33:50 1999 Geoff Hutchison * htnotify/htnotify.cc: Add copyright information. * htsearch/* htfuzzy/*: Ditto. Fri Sep 10 15:24:44 1999 Loic Dachary * htdig/Retriever.cc: change static WordList words to object member. words.Close() at end of Start function to make sure data is flushed by database. * htcommon/WordList.cc (Close): test isopen to prevent ugly crash. Remove isopen test in calling functions. Fri Sep 10 13:45:53 1999 Loic Dachary * htcommon/WordList.h htcommon/WordList.cc: methods Collect and Walk that factorise the behaviour of operator [], Prefix and WordRefs. * htcommon/WordList.h htcommon/WordList.cc: method Dump to dump an ascii version of the word database. * htcommon/WordReference.h,htcommon/WordReference.cc: method Dump to write an ascii version of a word. * htdig/htdig.cc: -t now also dump word database in ascii as well. * htdoc/attrs.html,cf_byprog.html,cf_byname.html: added doc for word_dump Thu Sep 9 20:30:18 1999 Geoff Hutchison * htfuzzy/Fuzzy.h, htfuzzy/Fuzzy.cc, htfuzzy/Prefix.cc, htfuzzy/Regex.cc, htfuzzy/Speling.cc, htfuzzy/Substring.cc, htfuzzy/htfuzzy.cc, htfuzzy.h: Change to use WordList code instead of direct access to the database. Thu Sep 9 14:55:59 1999 Gilles Detillieux * contrib/parse_doc.pl: fix bug in pdf title extraction. Tue Sep 7 23:49:41 1999 Geoff Hutchison * htdig/ExternalParser.h, htdig/ExternalParser.cc (parse): Change parsing of location to allow phrase searching -- location is *not* just 0-1000. * htdig/Plaintext.h, htdig/Plaintext.cc, htdig/PDF.cc: Ditto. * htdig/Retriever.h, htdig/Retriever.cc: Don't call HtStripPunctuation. This is now done in the WordList::Word method. * htcommon/WordList.h htcommon/WordList.cc (Prefix): New method to do prefix retrievals. Essentially the same as [], except the loop is broken only in the unlikely event that we retrieve something beyond the range set. (Exists): New method for checking the existance of a string--attempt to retrieve it and determine if anything's actually there. (Word): Call HtStripPunctuation as part of the cleanup. Tue Sep 7 21:37:44 1999 Geoff Hutchison * htcommon/defaults.cc: Add new configuration option removed_unretrieved_urls to remove docs that have not been accessed. * htmerge/docs.cc (convertDocs): Use it. * htcommon/defaults.h, htcommon/WordRecord.h, htcommon/WordReference.h: Add copyright notice to head of file. Mon Sep 6 10:32:59 1999 Geoff Hutchison * htlib/HtZlibCodec.h, htlib/HtZlibCodec.cc(instance): New method as used in other codecs. (encode, decode): Fix compilation errors. * htlib/Makefile.am: Added HtZlibCodec.cc to the compilation list. * htcommon/DocumentDB.cc (ReadExcerpt): Call HtZlibCodec to decompress the excerpt. (Add): Call HtZlibCodec to compress the excerpt before storing. (Open, Read): If the databases are already open, close them first in case we're opening under a different filename. (CreateSearchDB): Remove call to external sort program. Database is already sorted by DocID. * configure.in, configure: Remove check for external sort program. No longer necessary. * */Makefile.in: Regenerate using automake. Sun Sep 5 13:50:34 1999 Geoff Hutchison * htmerge/docs.cc: Ensure a document with empty excerpt has actually been retrieved. Otherwise document stubs are always removed. * htlib/String.cc: Implement the nocase_compare method. * htcommon/WordReference.cc: Implement a compare method for WordRefs to use in sorting. Uses the above. * htcommon/DocumentRef.h, htcommon/DocumentRef.cc: Update the headers. * htcommon/DocumentDB.h: Ditto. Sun Sep 5 01:37:27 1999 Geoff Hutchison * htcommon/WordList.cc(Flush): Call Add() instead of storing the data ourselves. Additionally, don't open the database ourself (and then close it), instead call Open() if it's not open already. * htcommon/DocumentRef.h, htcommon/DocumentRef.cc(AddDescription): Pass in a WordList to use when adding link text words. Ensures that the word db is never opened twice for writing. * htdig/Retriever.cc: Call AddDescription as above. * htdig/Server.cc(ctor): If debugging, write out an entry for the robots.txt file. * htlib/HtHeap.cc(percolateUp): Fix a bug where the parent was not updated when moving up more than once. (pushDownRoot): Fix a bug where the root was inproperly pushed down when it required looping. Fri Sep 3 16:23:23 1999 Geoff Hutchison * htlib/HtHeap.cc(Remove): Correct bug where after a removal, the structure was not "re-heapified" correctly. The last item should be moved to the top and pushed down. (pushDownRoot): Don't move items past the size of the underlying array. * htdig/Server.h, htdig/Server.cc: Change _paths to work on a heap, based on the hopcount. Ensures on a given server that the indexing will be done in level-order by hopcount. Wed Sep 01 15:40:37 1999 Loic Dachary * test: implement minimal tests for htsearch and htdig Tue Aug 31 02:17:04 1999 Geoff Hutchison * htcommon/WordRecord.h: Change back to struct to ensure integrity when compressed and stored in the word database. * htcommon/WordList.cc (Flush): Use HtPack to compress the WordRecord before storage. ([], WordRefs): Use HtUnpack to decompress the WordRecord after storage. Sun Aug 29 00:42:07 1999 Geoff Hutchison * htsearch/htsearch.cc (convertToBoolean): Remove debugging strings. * htsearch/parser.h: Add new method score(List) to merge scoring for both standard and phrase searching. * htsearch/parser.cc(phrase): Keep the current list of successful matched words around to pass to score and perform_phrase. (perform_phrase): Naively (and slowly, but correctly) loop through past words to make sure they match DocID as well as successive locations. Move scoring to score(). (perform_push): Move scoring to score(). (score): Loop through a list of WordReferences and create a list of scored DocMatches. Sun Aug 29 00:33:17 1999 Geoff Hutchison * htsearch/htsearch.cc(createLogicalWords): Hack to produce correct output with phrase searching (e.g. anything in quotes is essentially left alone). Ensure the StringMatch pattern includes the phrase with correct spacing as well. (setupWords): Add a " token whenever it occurs in the query. (convertToBoolean): Make sure booleans are not inserted into phrases. * htsearch/parser.h: Add new methods phrase and perfor_phrase to take care of parsing phrases and performing the actual matching. * htsearch/parser.cc(lexan): Return a '"' when present for phrase searching. (factor): Call phrase() before parsing a factor--phrases are the highest priority, so ("RedHat Linux" & Debian) ! Windows makes sense. (phrase): New method--slurps up the rest of a phrase and calls perform_phrase to do the matching. (perform_phrase): New method--currently just calls perform_and to give the simulation of a phrase match. Sat Aug 28 15:57:53 1999 Geoff Hutchison * htdig/Server.h, htdig/Server.cc: Undo yesterdays change -- still very buggy and shouldn't be used yet. * htdig/Retriever.cc (parse_url): Change default index to 1 to more closely match DocIDs shown with verbose output. * htsearch/DocMatch.h: Change score to double and clean up headers. * htcommon/WordRecord.h: Change unnecessary long ints (id and flags) to plain ints. * htdig/HTML.cc (parse): Call got_word with actual word sequence (i.e. 1, 2, 3...) rather than scaling to 1-1000 by character offset. * htlib/Database.h, htlib/DB2_db.h, htlib/DB2_hash.h: Change Get_Item to Get_Next(String item) to return the data as a reference. This makes it easier to use in a loop and cuts the database calls in half. * htlib/DB2_db.cc, htlib/DB2_hash.cc: Implement it, making sure we keep the possibly useful data around, rather than tossing it! * htsearch/htsearch.cc(htsearch): Don't attempt to open the word db ourselves. Instead, pass the filename off to the parser, which will do it through WordList. * htsearch/parser.h: Use a WordList instead of a generic Database. * htsearch/parser.cc(perform_push): Use the WordList[] operator to return a list of all matching WordRefs and loop through, summing the score. * htcommon/WordList.cc (Flush): Don't use HtPack on the data--somehow when unpacking, there's a mismatch of sizes. (Read): Fix thinko where we attempted to open the database as a DB_HASH. ([]): Don't use HtUnpack since we get mismatches. Use the new Get_Next(data) call instead of calling Get_Item separately. (WordRefs): Same as above. Fri Aug 27 09:44:09 1999 Geoff Hutchison * htdig/Retriever.cc (Need2Get): Remove duplicate detection code for local_urls. The code is somewhat buggy and should be replaced by more general code shortly. * htdig/Server.h, htdig/Server.cc (push, pop): Change _paths to a HtHeap sorted on hopcount first (and order placed on heap second). Ensures that on each server, the order indexed is guaranteed to be level-order by hopcount. * htdig/URLRef.h, htdig/URLRef.cc (compare): Add comparison method to enable sorting by hopcount. Fri Aug 27 09:36:35 1999 Geoff Hutchison * htcommon/WordList.h, htcommon/WordList.cc (WordList): Change words to a list instead of a dictionary for minor speed improvement. Thu Aug 26 11:18:20 1999 Gilles Detillieux * htcommon/defaults.cc, htdoc/attrs.html: increase default maximum_word_length to 32. Wed Aug 25 16:50:16 1999 Gilles Detillieux * htdig/Retriever.cc(got_word): add code to check for compound words and add their component parts to the word database. * htdig/PDF.cc(parseString), htdig/Plaintext.cc(parse): Don't strip punctuation or lowercase the word before calling got_word. That should be left up to got_word & Word methods. * htlib/StringMatch.h, htlib/StringMatch.cc(Pattern, IgnoreCase): Add an IgnorePunct() method, which allows matches to skip over valid punctuation, change Pattern() and IgnoreCase() to accomodate this. * htsearch/htsearch.cc(main, createLogicalWords): use IgnorePunct() to highlight matching words in excerpts regardless of punctuation, toss out old origPattern, and don't add short or bad words to logicalPattern. * htlib/HtWordType.h, htlib/HtWordType.cc(Initialize): set up and use a lookup table to speed up HtIsWordChar() and HtIsStrictWordChar(). Mon Aug 23 10:13:05 1999 Gilles Detillieux * htdig/HTML.cc(parse): fix problems with null pointer when attempting SGML entity decoding on bare &, as reported by Vadim Chekan. Thu Aug 19 11:52:06 1999 Gilles Detillieux * htsearch/htsearch.cc(main): Fix to allow multiple keywords input parameter definitions. * contrib/parse_doc.pl: make spaces optional in LANGUAGE = POSTSCRIPT PJL test. Wed Aug 18 11:27:46 1999 Gilles Detillieux * htdig/PDF.cc(parse): Fixed wrong variable name in new code. Double-Oops! (It was Friday the 13th, after all...) Tue Aug 17 16:26:46 1999 Gilles Detillieux * htlib/HtHeap.cc(Remove): apply Geoff's patch to fix Remove. * htlib/HtVector.h, htlib/HtVector.cc(Index): various bounds overrun bug fixes and checking in Last(), Nth() & Index(). Mon Aug 16 13:55:10 1999 Gilles Detillieux * htsearch/Display.cc(expandVariables): fix up test for & Mon Aug 16 12:08:57 1999 Gilles Detillieux * Makefine.am, Makefile.in, installdir/Makefile.am, installdir/Makefile.in: change all remaining INSTALL_ROOT to DESTDIR. Fri Aug 13 15:44:31 1999 Gilles Detillieux * htdig/PDF.cc(parse): added missing ')' in new code. Oops! * htlib/strptime.c, htlib/mktime.c: added #include "htconfig.h" to pick up definitions from configure program. Let's try to remember that config.h != htconfig.h! Fri Aug 13 14:49:07 1999 Loic Dachary * configure.in: removed unused HTDIG_TOP, changed AM_WITH_ZLIB by CHECK_ZLIB Fri Aug 13 14:00:16 1999 Gilles Detillieux * htdig/PDF.cc(parse), htcommon/defaults.cc, htdoc/attrs.html (pdf_parser): Removed -pairs option from default arguments, added special test for acroread to decide whether to use output file or directory as last argument (also adds -toPostScript if missing). Program now tries to test for existance of parser before trying to call it. Fri Aug 13 10:10:16 1999 Gilles Detillieux * htdoc/attrs.html(pdf_parser): updated xpdf version number. Thu Aug 12 17:09:37 1999 Gilles Detillieux * contrib/parse_doc.pl: updated for xpdf 0.90, plus other fixes. Thu Aug 12 11:12:07 1999 Gilles Detillieux * htdoc/attrs.html(logging): added Geoff's description of log lines. Thu Aug 12 11:21:12 1999 Loic Dachary * strptime fixes : AC_FUNC_STRPTIME defined in acinclude.m4 and used in configure.in, conditional compilation of strptime.c (only if HAVE_STRPTIME not defined), removed Htstrptime (strptime.c now defines strptime), changed all calls to Htstrptime to calls to strptime. Wed Aug 11 16:59:41 1999 Loic Dachary * */Makefile.am: use -release instead of -version-info because nobody wants to bother with published shared lib interfaces version numbers at present. * htlib/Makefile.am: added langinfo.h Wed Aug 11 15:00:07 1999 Loic Dachary * acconfig.h: removed MAX_WORD_LENGTH * re-run auto* to make sure chain is consistent * Makefile.am: improve distclean for tests Wed Aug 11 13:46:22 1999 Loic Dachary * configure.in: change --enable-test to --enable-tests so that Berkeley DB tests are not activated. Since they depend on tcl this can be a pain. * acinclude.m4: AM_PROG_TIME locate time command + find out if verbose output is -l (freebsd) or -v (linux) Wed Aug 11 13:13:39 1999 Loic Dachary * acinclude.m4 : AM_WITH_ZLIB autoconf macro for zlib detection that allows --with-zlib=DIR to specify the install root of zlib, --without-zlib to prevent inclusion of zlib. If nothing specified zlib is searched in /usr and /usr/local. --disable-zlib is replaced with --without-zlib. * configure.in,configure,aclocal.m4,db/dist/acinclude.m4, db/dist/aclocal.m4,db/dist/configure,db/dist/configure.in: changed to use AM_WITH_ZLIB Tue Aug 10 21:14:34 1999 Geoff Hutchison * htsearch/Display.cc (outputVariable): Fix compilation error with assignment between char * and char *. * htsearch/htsearch.cc (main): Use cleaner trick to sidestep discarding const char * as suggested by Gilles. Tue Aug 10 17:24:12 1999 Gilles Detillieux * htsearch/Display.cc(expandVariables): clean up, simplify and label lexical analyzer states. Tue Aug 10 17:04:54 1999 Gilles Detillieux * htsearch/Display.cc(expandVariables, outputVariable): add handling for $%(var) and $&(var) in templates. Still to be documented. Tue Aug 10 20:13:52 1999 Loic Dachary * db/mp/mp_bh.c: fixed HAVE_ZLIB -> HAVE_LIBZ Tue Aug 10 17:58:01 1999 Loic Dachary * configure,configure.in,db/dist/configure.in,db/dist/configure: added --with-zlib configure flag for htdig to specify zlib installation path. Motivated to have compatible tests between htdig and db as far as zlib is concerned. Otherwise configuration is confused and miss an existing libz. Tue Aug 10 17:44:49 1999 Loic Dachary * db/mp/mp_fopen.c: fixed cmpr_open called even if libz not here Tue Aug 10 17:40:53 1999 Loic Dachary * htlib/langinfo.h: header missing on FreeBSD-3.2, needed by strptime.c Tue Aug 10 11:43:14 1999 Gilles Detillieux * htdig/HTML.h, htdig/HTML.cc(parse, do_tag): fix problems with SGML entity decoding, add decoding of entities within tag attributes. Mon Aug 9 21:13:50 1999 Geoff Hutchison * htdig/HtHTTP.h(SetRequestMethod): Fix declaration to be void. * htdig/Transport.h(GetRequestMaxDocumentSize): Fix declaration to return int. * htdig/Retriever.cc(got_href): Fix mistake in hopcount calculations. Now returns the correct hopcount even for pages when a faster path is found. (Still need to change indexing to sort on hopcount). * htsearch/htsearch.cc(main): Fix compiler error in gcc-2.95 when discarding const by using strcpy. It's a hack, hopefully there's a better way. Mon Aug 9 17:23:15 1999 Gilles Detillieux * htlib/URL.cc(ServerAlias): fix small memory leak in new default path code (don't need to allocate new from string each time). * htlib/cgi.cc(init): Fix PR#572, where htsearch crashed if CONTENT_LENGTH was not set but REQUEST_METHOD was. * htfuzzy/Fuzzy.cc(getWords), htfuzzy/Metaphone.cc(vscode): Fix Geoff's change of May 15 to Fuzzy.cc, add test to vscode macro to stay in array bounds, so non-ASCII letters to cause segfault. Should fix PR#514. Mon Aug 9 17:03:45 1999 Gilles Detillieux * include/htconfig.h.in, htcommon/WordList.cc(Word,Flush&BadWordFile), htcommon/DocumentRef.cc(AddDescription), htcommon/defaults.cc, htsearch/parser.cc(perform_push), htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Convert the MAX_WORD_LENGTH compile-time option into the run-time configuration attribute maximum_word_length. This required reinserting word truncation code that had been taken out of WordList.cc. Mon Aug 9 16:34:14 1999 Gilles Detillieux * htdig/HtHTTP.cc (isParsable): allow application/pdf as parsable, to use builtin PDF code. * htdig/HtHTTP.cc (ParseHeader), htdig/Document.cc (readHeader): clean up header parsing. * htdig/Document.cc (getdate): make tm static, so it's initialized to zeros. Should fix PR#81 & PR#472, where strftime() would crash on some systems. Idea submitted by benoit.sibaud@cnet.francetelecom.fr * htlib/URL.cc (parse): fix PR#348, to make sure a missing or invalid port number will get set correctly. Mon Aug 9 15:42:41 1999 Gilles Detillieux * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Added descriptions for attributes that were missing, added a few clarifications, and corrected a few defaults and typos. Covers PR#558, PR#626, and then some. * configure.in, configure, include/htconfig.h.in, htlib/regex.c: PR#545 fixed - configure tests for presence of alloca.h for regex.c Sat Aug 07 13:40:17 1999 Loic Dachary * configure.in: remove test for strptime. Run autoconf + autoheader. * htlib/HtDateTime.cc: always use htdig strptime, do not try to use existing function in libc. * htlib/HtDateTime.h: move inclusion of htconfig.h on top of file, change #ifdef HAVE_CONFIG to HAVE_CONFIG_H Fri Aug 6 16:37:33 1999 Gilles Detillieux * htdig/Document.cc (UseProxy): fix call to match() and test of return value to work as documented for http_proxy_exclude (PR#603). Fri Aug 06 15:06:23 1999 * db/dist/config.hin, db/mp/mp_cmpr.c db/db/db.c, db/mp/mp_fopen.c: disable compression if zlib not found by configure. Thu Aug 05 12:27:15 1999 * test/dbbench.cc: invert -z and -Z for consistency * test/Makefile.am: add dbbench call examples Thu Aug 05 11:38:58 1999 Loic Dachary * test/Makefile.am: all .html go in distribution, compile dbbench that tests Berkeley DB performances. * configure.in/Makefile.am: conditional inclusion of the test directory in the list of subdirs (--enable-test). The list of subdirs is now @HTDIGDIRS@ in configure.in & Makefile.am * db/*: Transparent I/O compression implementation. Defines the DB_COMPRESS flag. For instance DB_CREATE | DB_COMPRESS. * db/db_dump/load: add -C option to specify cache size to db_dump/db_load Wed Aug 4 22:57:27 1999 Geoff Hutchison * db/*: Import of Sleepycat's Berkeley DB 2.7.5. Wed Aug 4 22:40:49 1999 Geoff Hutchison * contrib/htparsedoc/htparsedoc: Add in contributed bug fixes from Andrew Bishop to work on SunOS 4.x machines. Wed Aug 4 01:58:52 1999 Geoff Hutchison * COPYING, htdoc/COPYING, configure.in, Makefile.am, Makefile.in: Update information to use canonical version of the GPL from the FSF. In particular, this version has the correct mailing address of the FSF. Mon Aug 02 11:28:00 1999 Gabriele Bartolini * htlib/htString.h, htlib/String.cc : added the possibility to insert an unsigned int into a string. * htdig.cc : with verbose mode shows start and end time. Thu Jul 22 18:10:00 1999 Gabriele Bartolini * htdig/Transport.cc, htdig/HtHTTP.cc : modified the destructors. Thu Jul 22 13:10:00 1999 Gabriele Bartolini * htdig/Transport.cc, htdig/Transport.h, htdig/HtHTTP.cc, htdig/HtHTTP.h: Re-analyzed inheritance methods and attributes of the 2 classes. This is a first step, not definitive ... cos it still doesn't work as I hope. Tue Jul 20 11:21:52 1999 * configure.in : added AM_MAINTAINER_MODE to prevent unwanted dependencies check by default. * db/Makefile.in : remove Makefile when distclean Mon Jul 19 13:23:53 1999 * Makefile.config (INCLUDES): added -I$(top_srcdir)/include because automatically -I../include is not good, added -I$(top_builddir)/db/dist because some db headers are configure generated (if building in a directory that is not the source directory). * rename db/Makefile db/Makefile.in: otherwise it does not show up if if building in a directory that is not the source directory. Mon Jul 19 13:02:22 1999 * .cvsignore: do not ignore Makefile.config Sun Jul 18 22:47:49 1999 Geoff Hutchison * htsearch/parser.cc: Eliminated compiler errors. Currently returns no matches until bugs in the WordList code are fixed. Sun Jul 18 22:42:04 1999 Geoff Hutchison * htmerge/htmerge.h: Cleanup, including WordRecord and WordReference as needed. * htmerge/htmerge.cc: Update for files necessary for merge calls. Call convertDocs before mergeWords so that the discardList gets the list of documents deleted. * htmerge/docs.cc: Update for difference in calling order. * htmerge/words.cc: Update (and significant cleanup) since WordList writes directly to db.words.db. Iterate over the stored words, deleting those from deleted documents. * htmerge/db.cc: Update to eliminate compiler errors. Currently disabled until bugs in the words code are fixed. Sun Jul 18 22:33:49 1999 Geoff Hutchison * htcommon/defaults.cc: Collapse the multiple heading_factors into one. (It's prohibitive to define a flag for each h* tag). Add a new url_factor for the text of URLs (presently unused). * htcommon/DocumentRef.cc(AddDescription): Use FLAG_LINK_TEXT as defined in htcommon/WordRecord.h. * htdig/Retriever.h: Change factor to accomodate flags instead of weighting factors. * htdig/Retriever.cc: Update to use flags, and define the indexed flags in factor as appropriate. * htdig/HTML.cc: Update calls to got_word with appropriate new offsets into factor[]. Sun Jul 18 22:18:16 1999 Geoff Hutchison * htcommon/WordReference.h, htcommon/WordRecord.h: Update to use flags instead of weight. * htcommon/WordList.h, htcommon/WordList.cc: Add database access routines to match DocumentDB.cc. (Word): Recognize flags instead of weight, simply add the word. (Duplicates expected!) (mark*): Simply delete the list of words. (flush): Rather than dump to a text file, dump directly to the db. Sun Jul 18 21:50:04 1999 Geoff Hutchison * htlib/Database.h, htlib/DB2_db.h, htlib/DB2_hash.h: Add new method Get_Item to access the data of the current item when using Get_Next() or Get_Next_Seq(). * htlib/DB2_db.h, htlib/DB2_hash.cc: Implement Get_Item() using cursor access. Sat Jul 17 12:59:01 1999 Geoff Hutchison * test/*.html: Added various HTML files as the beginnings of a testing suite. Fri Jul 16 16:06:27 1999 Loic Dachary * All libraries (except db) use libtools. Shared libraries are generated by default. --disable-shared to get old behaviour. Libraries are installed in all cases. * Change structure of default installation directory (match standard). database : var/htdig programs : bin libraries : lib Like default apache: conf : conf htdocs : htdocs/htdig cgi-bin : cgi-bin * Switch all Makefile.in into Makefile.am * CONFIG.in CONFIG : removed. Replaced with --with- arguments in configure.in * Makefile.config.in removed, only keep Makefile.config : automake automatically defines variables for each AC_SUBST variables. Makefile.config has HTLIBS + DEFINES * db/Makefile : added to forward (clean all distclean) targets to db/dist and implement distdir target. * acconfig.h : created to allow autoheader to work (contains GETPEERNAME_LENGTH_T HAVE_BOOL, HAVE_TRUE, HAVE_FALSE, NEED_PROTO_GETHOSTNAME). Extra definitions added before @TOP@ (TRUE, FALSE, VERSION, MAX_WORD_LENGTH, LOG_LEVEL, LOG_FACILITY). * installdir/Makefile.am : installation rules moved from Makefile.am to installdir/Makefile.am * include/Makefile.am : distribute htconfig.h.in and stamp-h.in * Makefile.am : do not pre-create the directories, creation is done during the installation * configure.in: CF_MAKE_INCLUDE not needed anymore : automake handles the include itself. Fri Jul 16 13:04:27 1999 Gilles Detillieux * htdig/HTML.cc(parse): fix to prevent closing ">" from being passed to do_tag(). Thu Jul 15 21:25:12 1999 Geoff Hutchison * htdig/Document.cc (readHeader, getParsable): Add back application/pdf to use builtin PDF code. * htdig/Makefile.in: Remove broken Postscript parser as it never worked. * htlib/URL.cc (normalizePath, path): Use config.Boolean as pointed out by Gilles. Thu Jul 15 15:54:30 1999 Gilles Detillieux * htdoc/attrs.html(pdf_parser & external_parsers): add corrections & clarifications, links to relevant FAQ entries. Thu Jul 15 18:00:00 1999 CEST Gabriele Bartolini * htlib/HtDateTime.cc, htlib/HtDateTime.h : added the possibility to initialize and compares HtDateTime with integers. Added the constructor HtDateTime (int) and various operator overloading methods. Wed Jul 14 22:57:14 1999 Geoff Hutchison * htlib/URL.cc (normalizePath, path): If not case_sensitive, lowercase the URL. Should ensure that all URLs are appropriately lowercased, regardless of where they're generated. Wed Jul 14 22:37:47 1999 Geoff Hutchison * htlib/DB2_db.cc (OpenReadWrite, OpenRead): Add flag DB_DUP to database to allow storage of duplicate keys (in this case, words). Tue Jul 13 15:36:40 1999 Gilles Detillieux * htdig/HTML.cc (do_tag): Fix handling of and , to use href= instead of src=. Mon Jul 12 22:31:48 1999 Hanno Mueller * contrib/scriptname/results.shtml: Remove unintentional $(VERSION). Mon Jul 12 22:20:40 1999 Geoff Hutchison * htdig/HTML.cc (do_tag): Cleanups suggested by Gilles, combining and , and and moving to a separate case. Sun Jul 11 19:32:38 1999 Hanno Mueller * contrib/README: Add scriptname directory. * contrib/scriptname/*: An example of using htsearch within dynamic SSI pages * htcommon/defaults.cc: Add script_name attribute to override SCRIPT_NAME CGI environment variable. * htdoc/FAQ.html: Update question 4.7 based on including htsearch as a CGI in SSI markup. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html, htdoc/hts_templates.html: Update based on behavior of script_name attribute. * htsearch/Display.cc: Set SCRIPT_NAME variable to attribute script_name if set and CGI environment variable if undefined. Sat Jul 10 00:22:34 1999 Geoff Hutchison * htfuzzy/Regex.cc (getWords): Anchor the match to the beginning of string, add regex-interpeted characters to extra_word_chars temporarily, and strip remaining punctuation before making a match. Fri Jul 9 22:35:57 1999 Geoff Hutchison * htsearch/Display.cc: Back out change of June 24. * htsearch/htsearch.cc: Ditto. * htsearch/htsearch.cc (setupWords): Remove HtStripPunctuation in favor of requiring Fuzzy classes to strip whatever punctuation is necessary. * htfuzzy/Fuzzy.h: Add HtWordType.h to #includes and update comments. * htfuzzy/Synonym.cc, htfuzzy/Substring.cc, htfuzzy/Speling.cc, htfuzzy/Prefix.cc, htfuzzy/Exact.cc, htfuzzy/Endings.cc, htfuzzy/Fuzzy.cc (getWords): Call HtStripPunctuation on input before performing fuzzy matching. Thu Jul 8 21:28:44 1999 Geoff Hutchison * htdig/HTML.cc (do_tag): Add support for parsing tags. Mon Jul 5 16:53:23 1999 Geoff Hutchison * htdig/htdig.cc (main): Insert '*' instead of username/password combination to hide credentials in process accounting. Sat Jul 3 17:35:52 1999 Geoff Hutchison * htdig/Transport.h(ConnectionWrite): Return value from Connection::write call. * htdig/URLRef.h, htdig/URLRef.cc: Cleanup and made hopcount default consistent with 7/3 change to DocumentRef.cc * htdig/Server.h, htdig/Server.cc, htdig/Retriever.cc: Cleanup and fixes to match URLRef calling interface. Sat Jul 3 16:37:29 1999 Geoff Hutchison * htdig/HTML.cc (do_tag): Fix robots parsing to allow multiple directives to work correctly. Fixes PR#578, as provided by Chris Liddiard . Sat Jul 3 00:47:51 1999 Geoff Hutchison * htdig/Makefile.in: Remove old SGMLEntities code. Sat Jul 3 00:26:55 1999 Geoff Hutchison * htcommon/DocumentRef.cc (Clear): Change default value of docHopCount to 0 to fix several hopcount bugs. * htdig/Transport.h, htdig/Transport.cc: Changes to support URL referers as well as authentication credentials. * htdig/HtHTTP.h, htdig/HtHTTP.cc(SetCredentials): Implement HTTP Basic Authentication credentials. (SetRequestCommand): Use Referer and Authentication headers if supplied. Sun Jun 30 11:26:00 1999 Gabriele Bartolini * htdig/Transport.h: Inserted the methods declarations regarding the connection management. The code has been moved out from the HtHTTP.h code. Also moved here the static variable 'debug'. * htdig/Transport.cc: Definition of the connection management code. The code has been moved out from the HtHTTP.cc code. * htdig/HtHTTP.h: Eliminated the connection management code and the static variable 'debug'. Inserted the 'modification_time_is_now' as a static variable, in order to respect the encapsulation principle. * htdig/HtHTTP.cc: Eliminated the connection management code and the static variable 'debug' initialization. Inserted the 'modification_time_is_now' initialization. Sun Jun 27 16:29:49 1999 Geoff Hutchison * htdig/HTML.h: Cleanup. * htcommon/defaults.cc: Added default for img_alt_factor for text weighting on ...' in string passed to do_tag. (do_tag): Index IMG ALT text. Fri Jun 25 17:58:44 1999 Geoff Hutchison * htdig/Transport.h: Fix virtual methods for Transport_Response to have defaults. * htdig/HtHTTP.h: Fix class declaration of HtHTTP class to prevent syntax error. Pointed out by Gabriele. * htdig/Transport.cc: Add (empty) ctor and dtor functions for Transport_Response. Thu Jun 24 22:28:44 1999 Geoff Hutchison * htsearch/htsearch.cc (main): Add support for form inputs configdir and commondir as contributed by Herbert Martin Dietze . * htsearch/Display.cc (createURL): If configdir and commondir are defined, add them to URLs sent for other pages. Wed Jun 23 23:00:18 1999 Geoff Hutchison * htdig/HtHTTP.h, htdig/HtHTTP.cc: Make a subclass of Transport. Wed Jun 23 22:08:20 1999 Geoff Hutchison * htlib/Configuration.cc (Add): Handle single-quoted values for attributes. Tue Jun 22 23:35:39 1999 Geoff Hutchison * htdig/Transport.h, htdig/Transport.cc: Virtual classes to handle transport protocols such as HTTP, FTP, WAIS, gopher, etc. * htdig/Makefile.in: Make sure they're compiled (not that there's much!) * htdig/HtHTTP.h: Add htdig.h to ensure config is defined. Mon Jun 21 14:33:10 1999 Gilles Detillieux * htdig/Document.cc(readHeader), htdig/HtHTTP.cc(ParseHeader): fix handling of modification_time_is_now in readHeader, add similar code to ParseHeader. Sun Jun 20 21:25:15 1999 Geoff Hutchison * htdig/Retriever.h: Add hop parameter to got_href method. Defaults to 1. * htdig/Retriever.cc(got_href): Use it instead of constant 1. * htdig/HTML.cc (do_tag): Use new hop parameter to keep the same hopcount for frame, embed and object tags. * htdig/Makefile.in: Make sure HtHTTP.cc is compiled. * htdig/HtHTTP.cc (ctor): Add default value for _server to make prevent strange segmentation faults. Fri Jun 18 09:53:30 1999 Gilles Detillieux * htcommon/DocumentRef.h, htcommon/DocumentRef.cc(Clear, Deserialize): add docHeadIsSet field, code for setting and getting it. * htcommon/DocumentDB.cc(Add): only put out excerpt record if DocHead is really set. * htmerge/doc.cc(convertDocs): add missing else after code to delete documents with no excerpts. (All these changes fix the disappearing excerpts problem in 3.2.) Wed Jun 16 23:04:38 1999 Geoff Hutchison * htdig/Document.cc (UseProxy): Change http_proxy_exclude to an escaped regex string. Allows for much more complicated rules. Wed Jun 16 16:04:07 1999 Gilles Detillieux * Makefile.config.in: fix typo in name IMAGE_URL_PREFIX. * htdig/Retriever.cc(IsValidURL): change handling of valids to only reject if list is not empty, give different error message. Wed Jun 16 14:40:56 1999 Gilles Detillieux * htsearch/htsearch.cc(main): pass StringList args to setEscaped() instead of unprocessed input[] char *'s. * htsearch/Display.cc(buildMatchList): cast score to (int) in maxScore calculation, to avoid compiler warnings. * htdig/htdig.cc(main): change comparison on minimalFile to avoid compiler warnings. Wed Jun 16 11:30:23 1999 Gilles Detillieux * htlib/HtRegex.cc(setEscaped): Fix appending of substring to avoid compiler warnings. * htlib/HtDateTime.cc(SettoNow): Strip out all the nonsense that doesn't work, set Ht_t directly instead. Wed Jun 16 09:58:12 1999 Gilles Detillieux * configure.in, configure, Makefile.config.in: Correct handling of SEARCH_FORM variable, as Gabriele recommended. Wed Jun 16 09:32:06 1999 Gilles Detillieux * htlib/cgi.h, htlib/cgi.cc(cgi & init), htsearch/htsearch.cc (main & usage): allow a query string to be passed as an argument. Wed Jun 16 08:43:09 1999 Gilles Detillieux * htcommon/Makefile.in, htdig/Makefile.in, htfuzzy/Makefile.in, htmerge/Makefile.in, htnotify/Makefile.in: Use standard $(bindir) variable instead of $(BIN_DIR). Allows for standard configure flags to set this. (Completes Geoff's change on May 15.) Tue Jun 15 14:31:50 1999 Gilles Detillieux * htdig/PDF.cc(parseNonTextLine): move line that clears _parsedString, so title cleared even if rejected. * htsearch/Display.cc(buildMatchList & sort): move maxScore calculation from sort to buildMatchList, so it's done even if there's only 1 match. Mon Jun 14 15:01:07 1999 Gilles Detillieux * htdig/Document.cc(RetrieveHTTP): Show "Unknown host" message if Connection::assign_server() fails (due to gethostbyname() failure). Mon Jun 14 13:52:34 1999 Gilles Detillieux * htcommon/defaults.cc, htsearch/Display.h, htsearch/Display.cc, htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html, htdoc/hts_templates.html: add template_patterns attribute, to select result templates based on URL patterns. Sun Jun 13 16:29:19 1999 Geoff Hutchison * htdig/Retriever.cc (IsValidURL): Add valid_extension list, as requested numerous times. * htcommon/defaults.cc: Add config attribute valid_extensions, with default as empty. Sat Jun 12 23:10:39 1999 Geoff Hutchison * htcommon/DocumentRef.h: Fix thinkos introduced in change earlier today. Actually compiles correctly now. Sat Jun 12 22:37:22 1999 Geoff Hutchison * htdig/HtHTTP.cc (ParseHeader): Fix parsing to take empty headers into account. Fixes PR#557. * htsearch/Display.h, htsearch/Display.cc (excerpt): Fix declaration to refer to first as reference--ensures ANCHOR is properly set. Fixes PR#541 as suggested by . * htfuzzy/Endings.cc (getWords): Fixed PR#560 as suggested by Steve Arlow . Solves problems with fuzzy matching on words like -ness: witness, highness, likeness... Tries to interpret words as root words before attempting stemming. * installdir/search.html (Match): Add Boolean to default search form, as suggested by PR#561. * htlib/URL.cc (URL): Fix PR#566 by setting the correct length of the string being matched. 'http://' is 7 characters... Sat Jun 12 19:06:36 1999 Geoff Hutchison * htlib/HtZlibCodec.h, htlib/HtZlibCodec.cc: New files. Provide general access to zlib compression routines when available. * htcommon/DocumentRef.h, htcommon/DocumentRef.cc: Remove compression access and restore DocHead access through default methods. Compression of excerpts will occur through the HtZlibCodec classes and through the DocumentDB excerpt access. Sat Jun 12 15:25:08 1999 Geoff Hutchison * htmerge/docs.cc (convertDocs): Load excerpt from external database before considering it empty. Sat Jun 12 14:41:54 1999 Geoff Hutchison * htsearch/Display.cc (displayMatch): Added patch from Torsten Neuer to fix PR# 554. * htdig/HTML.cc (do_tag): Add parsing for and , including suggestions from Gilles as to condensing cases with parsing. Sat Jun 12 14:00:39 1999 Geoff Hutchison * htdig/ExternalParser.cc (parse): Quote the filename before passing it to the command-line to prevent shell escapes. Fixes PR#542. Fri Jun 11 15:59:10 1999 Gilles Detillieux * htlib/URL.cc(removeIndex): use CompareWord instead of FindFirstWord, to avoid substring matches. Wed Jun 2 15:51:00 1999 Gilles Detillieux * htlib/URLTrans.cc(encodeURL): Fix to ensure that non-ASCII letters get URL-encoded. Mon May 31 22:40:29 1999 Geoff Hutchison * htcommon/DocumentDB.cc(ReadExcerpt): Fix silly typos with methods, thinko with docID. (Add): Add the excerpt *before* the URL index is written. * htdig/Retriever.cc(isValidURL): Remove code restricting URLs to relative and http://. * htdig/htdig.cc(main): Unlink the doc_excerpt file when doing an initial dig. (main): Fix silly typo with minimumFile. * htmerge/db.cc(mergeDB): Call DocumentDB::Open() with doc_excerpt for consistency--doesn't actually do anything with it. * htmerge/docs.cc(convertDocs): Ditto. Also don't delete a document simply because it has an empty DocHead. Excerpts are now stored in a separate database! * htmerge/htmerge.h: Call mergeDB and convertDocs with doc_excerpt parameter. * htmerge/htmerge.cc(main): Ditto. * htsearch/Display.h: Call ctor with all three doc db filenames. * htsearch/Display.cc(Display): Call DocumentDB::Open with above. (excerpt): Retrieve the excerpt from the excerpt database. * htsearch/htsearch.cc: Call Display::Display with all three doc db filenames. Mon May 31 15:08:30 1999 Geoff Hutchison * htcommon/DocumentDB.h: Add new method ReadExcerpt to read the excerpt from the separate (new) excerpt database. Change Open() and Read() methods to account for this new database. * htcommon/DocumentDB.cc (Open): Open the excerpt database too. (Read): Ditto. (Close): Close it if it exists. (ReadExcerpt): Explicitly read the DocHead of this DocumentRef. (Add): Make sure DocHeads go into the excerpt database. (Delete): Make sure we delete the associated excerpt too. (CreateSearchDB): Make sure we grab the excerpt from the database. * htcommon/DocumentRef.cc(Serialize): Don't serialize the DocHead field, this is done in the DocumentDB code. * htcommon/defaults.cc(modification_time_is_now): Set to true to avoid problems with not setting dates when no Last-Modified: header appears. (doc_excerpt): Add new attribute for the filename of the excerpt database. * htdig/HtHTTP.h: Remove incorrect virtual declarations from Request and EstablishConnection methods. Assign void return value to ResetStatistics since it doesn't return a value. * htdig/htdig.cc (main): Add new "minimal" flag '-m' to only index the URLs in the supplied file. Sets hopcount to ignore links. Sun May 30 19:36:15 1999 Alexander Bergolth * htlib/URL.cc (normalizePath): Fix bug that caused endless loops and core dumps when normalizing URLs with more than one of ( "/../" | "/./" | "//" | "%7E" ) * htlib/HtDateTime.cc (Httimegm): Call Httimegm in timegm.c unless HAVE_TIMEGM. Wed May 26 23:15:46 1999 Geoff Hutchison * htmerge/db.cc (mergeDB): Add patch contributed by Roman Dimov to fix problems with confusing docIDs, resulting in documents in main db removed when the corresponding DocID was supposed to be removed from the merged db. Wed May 26 11:30:22 1999 Geoff Hutchison * htsearch/Display.h, htsearch/Display.cc, htsearch/htsearch.cc: Switch restrict and excludes to use HtRegex instead of StringMatch. * htdig/htdig.cc (main): Fix typo clobbering setting of excludes. Obviously fixes problems with badquerystr and excludes! * htdig/HtHTTP.cc (ParseHeader): Change parsing to skip extra whitespace, as in 5/19 Document.cc(readHeader) change. Wed May 19 22:17:49 1999 Geoff Hutchison * htdig/HtHTTP.cc, htdig/HtHTTP.h: Add new files, contributed by Gabriele. A start at an HTTP/1.1 implementation. * htdig/Document.cc (readHeader): Fix change of 5/16 to actually work! :-) * htsearch/Display.cc (expandVariables): Change end-of-expansion test to include states 2 and 5 to ensure templates ending in } are still properly expanded, as suggested by Gilles. Mon May 17 14:31:31 1999 Geoff Hutchison * htlib/HtRegex.cc (setEscaped): Use full list of characters to escape as suggested by Gilles. Sun May 16 17:27:51 1999 Geoff Hutchison * htdig/Document.cc (readHeader): Since multiple whitespace characters are allowed after headers, don't use strtok. (readHeader): We no longer pretend to parse Word, PostScript, or PDF files internally. (getParsable): Don't generate PostScript or PDF objects since we no longer recommend using them. Sun May 16 17:07:19 1999 Geoff Hutchison * htlib/HtRegex.cc (setEscaped): Ensure escaping does not loop beyond the end of a string. * htdig/Retriever.cc (IsValidURL): Fix badquerystr parsing to use HtRegex as expected. (Oops!) * htdig/HTML.cc (parse): Use HtSGMLCodec during parsing, rather than encoding the whole document at the beginning. More consistent with previous use of SGMLEntities. Sat May 15 12:57:40 1999 Geoff Hutchison * htlib/URL.cc (normalizePath): Remove extra (useless) variable declarations. * htlib/htString.h, htlib/String.cc: Add new method Nth to solve problems with (String *)->[]. * htlib/HtRegex.h, htlib/HtRegex.cc: Added new method setEscaped(StringList) to produce a pattern connected with '|' of possibly escaped strings. Strings are not escaped if enclosed in [] and the brackets are removed from unescaped regex. * htdig/htdig.h: Use HtRegex instead of StringMatch for limiting by default. * htdig/Retriever.cc: As above. * htdig/htdig.cc(main): As above. Use setEscaped to set limits correctly (i.e. in a backwards-compatible way). Sat May 15 11:24:26 1999 Geoff Hutchison * htfuzzy/Speling.h, htfuzzy/Speling.cc: New files for simple spelling corection. Currently limited to transpostion and added character errors. Missing character errors to be added soon. * htfuzzy/Makefile.in: Compile it. * htfuzzy/Fuzzy.cc (getFuzzyByName): Use it. * htcommon/defaults.cc: Add new option minimum_speling_length for the shortest query word to receive speling fuzzy modifications. Should prevent problems with valid words generating unrelated "corrections" of words. Default is 5 chars. Sat May 15 11:18:27 1999 Geoff Hutchison * htfuzzy/Fuzzy.cc (getWords): Ensure word is not an empty or null string. * htfuzzy/Metaphone.cc (generateKey): Ditto. Should solve PR#514. * htdig/Document.cc (Reset): Do not use modification_time_is_now attribute. Simply reset modtime to 0, time is set elsewhere. * Makefile.config.in: Add options from separate CONFIG files. * configure.in, configure: Add configure-level switches for --with-image-url-prefix= and --with-search-form=. Do not generate CONFIG file (hopefully to be phased out soon). * */Makefile.in: Make linking CONFIG-dependent files depend on Makefile.config, not CONFIG. * Makefile.in: Use standard $(bindir) variable instead of $(BIN_DIR). Allows for standard configure flags to set this. Tue May 11 11:15:08 1999 Geoff Hutchison * htlib/HtDateTime.h, htlib/HtDateTime.cc: Updates from Gabriele, fixing SetToNow() and adding GetDiff to return the difference in time_t between two objects. * htdig/Retriever.cc (Need2Get): Add patch from Warren Jones to keep track of inodes on local files to eliminate duplicates. Hopefully this will serve for a first-try at a signature method for HTTP as well. Tue May 4 20:20:40 1999 Geoff Hutchison * htfuzzy/Regex.h, htfuzzy/Regex.cc: Add new regex fuzzy algorithm, based on Substring and Prefix. * htfuzzy/Fuzzy.cc (getFuzzyByName): Add it. * htfuzzy/Makefile.in: Compile it. * htcommon/defaults.cc: Add new attribute regex_max_words, same concept as substring_max_words. * htfuzzy/Exact.cc, htfuzzy/Substring.cc, htfuzzy/Prefix.cc: Define names attribute for debugging purposes. * installdir/htdig.conf: Fix the comments for search_algorithm to refer to all the current possibilities. * htlib/HtRegex.cc (match): Slight cleanup of how to return. Tue May 4 15:28:38 1999 Geoff Hutchison * htsearch/htsearch.cc (reportError): Add e-mail of maintainer to error message. Should help direct people to the correct place. * htdig/Retriever.cc (IsValidURL): Lowercase all extensions from bad_extensions as well as all extensions used in comparisons. Ensures we're using case-insenstive matching. Mon May 3 23:20:22 1999 Geoff Hutchison * htdig/Retriever.cc (IsValidURL): Fix typo with #else statement for REGEX. * htdig/htdig.cc: Add conditionals for REGEX to use HtRegex instead of StringMatch methods when defined. * htlib/HtDateTime.h: Update to remove definitions of true and false, established by May 2 change in include/htconfig.h.in as contributed by Gabriele. * htlib/HtDateTime.cc: Replace call to mktime internal function to Httimegm in timegm.c, contributed by Leo. * htlib/timegm.c: Declare my_mktime_gmtime_r to prevent compiler errors with incompatible gmtime structures, contributed by Leo. * configure.in: Rearrange date/time checks for clarity. * configure: Regenerate using autoconf. * include/htconfig.in: Add HAVE_STRFTIME flag. Sun May 2 18:49:04 1999 Alexander Bergolth * configure.in, include/htconfig.h.in: Added a configure test for the availability of the bool type. Fri Apr 30 20:00:09 1999 Geoff Hutchison * htlib/HtDateTime.h, htlib/HtDateTime.cc: Update with new versions sent by Gabriele. Fri Apr 30 19:30:42 1999 Geoff Hutchison * htlib/HtRegex.h, htlib/HtRegex.cc: New class, contributed by Peter D. Gray as a small wrapper for system regex calls. * htlib/Makefile.in: Build it. * htdig/htdig.h: Use it if REGEX is defined. * htdig/htdig.cc: Ditto. * htdig/Retriever.cc: Ditto. * htsearch/Display.cc(generateStars): Remove extra newline after STARSRIGHT and STARSLEFT variables, noted by Torsten Neuer . Fri Apr 30 18:52:56 1999 Alexander Bergolth * htlib/URL.cc(ServerAlias): port for server_aliases entries now defaults to 80 if omitted. Wed Apr 28 19:57:38 1999 Geoff Hutchison * htlib/HtDateTime.h, htlib/HtDateTime.cc: New class, contributed by Gabriele. * htlib/Makefile.in: Compile it. * README: Update message from 3.1.0 (oops!) to 3.2.0, remove rx directory. * installdir/htdig.conf: Add example of no_excerpt_show_top attribute in line with most user's expectations. * contrib/README: Mention contributed section of the website. * Makefile.in: Ignore mailarchive directory--now removed from CVS. Wed Apr 28 10:46:31 1999 Gilles Detillieux * htmerge/db.cc(mergeDB): fix a few errors in how the merge index name is obtained. Tue Apr 27 23:00:39 1999 Geoff Hutchison * Makefile.config.in: Remove now-useless LIBDIRS variable. * mailarchive/Split.java, mailarchive/htdig: Remove ancient mailarchive stuff. Tue Apr 27 18:01:52 1999 Gilles Detillieux * htsearch/Display.cc(setupImages): Remove code setting URLimage to a bogus pattern (remnant left over after merge). Tue Apr 27 16:43:08 1999 Gilles Detillieux * htdig/Document.cc(RetrieveHTTP): Show "Unable to build connection" message at lower debug level. Tue Apr 27 11:24:19 1999 Geoff Hutchison * htsearch/Display.h: Remove sort, compare functions re-introduced in merge. Moved to ResultMatch by Hans-Peter's April 19th chnages. * htsearch/Display.cc: Remove bogus call to ResultMatch:setRef, removed by Hans-Pater's April 19th changes. Sat Apr 24 21:08:35 1999 Geoff Hutchison * Merge in changes from 3.1.2 (see below). * htcommon/WordList.cc: Change valid_word to use iscntl(). * htdig/Plaintext.cc: Remove CVS Log. * htdig/Retriever.cc: Fix ancient bug with empty excludes list. * htlib/List.cc: Remove CVS Log, use more succinct test for out-of-bounds. * htsearch/Display.cc: Fix logic with starPatterns, only show top of META description. * htsearch/Display.h: Introduce headers needed for sort functionality. * installdir/htdig.conf: Add example max_doc_size attribute as well as example for including start_url from a file. * htdoc/ChangeLog, htdoc/RELEASE.html, htdoc/FAQ.html, htdoc/where.html, htdoc/cf_byname.html, htdoc/cf_byprog.html, htdoc/uses.html, htdoc/contents.html, htdoc/mailarchive.html: Merge in documentation updates from 3.1.2. Sat Apr 24 15:18:45 1999 Hans-Peter Nilsson * htsearch/Display.cc (sort): Return immediately if <= 1 items to sort. Mon Apr 19 00:53:06 1999 Hans-Peter Nilsson * htsearch/ResultMatch.h (create): New. All (the only) ctor caller changed to use this. (setRef, getRef): Removed. Callers changed to use nearby data. (incomplete): Removed. (setIncompleteScore): Renamed to... (setScore): ...this. All callers changed. (setSortType): New. (getTitle, getTime, setTitle, setTime, getSortFun): New virtual functions. (enum SortType): Moved from Display, private. (mySortType): New static member. * htsearch/ResultMatch.cc (mySortType): Define static member variable. (getScore): Remove handling of "incomplete". Moved to ResultMatch.h (getTitle, getTime, setTitle, setTime): New dummy functions. (class ScoreMatch, class TimeMatch, class IDMatch, class TitleMatch): Derived classes with compare functions (from Display) and extra sort-method-related members, as needed. (setSortType): New, mostly moved from Display. (create): New. * htsearch/Display.h: Changed first argument from ResultMatch * to DocumentRef *. (compare, compareTime, compareID, compareTitle, enum SortType, sortType): Removed. * htsearch/Display.cc (display): Call ResultMatch::setSortType and output syntax error page for invalid sort methods. (displayMatch): Change first argument from ResultMatch * to DocumentRef *ref. All callers changed. (buildMatchList): Remove call to sortType and typ variable. Always call (ResultMatch::)setTime and setTitle. Remove extra call to setID. (sort): Call (ResultMatch::)getSortFun for qsort compare function. (compare, compareTime, compareID, compareTitle, sortType): Removed. Wed Apr 14 21:21:35 1999 Alexander Bergolth * htlib/regex.c: fixed compile problem with AIX xlc compiler * htlib/HtHeap.h: fixed compile problem with AIX xlc compiler (bool) * htlib/HtVector.h: ditto * htsearch/Display.cc: fixed typo Wed Apr 14 00:17:06 1999 Geoff Hutchison * htsearch/Display.h: Add compareID for sorting results by DocID. * htsearch/Display.cc: As above. Tue Apr 13 23:50:28 1999 Geoff Hutchison * htcommon/defaults.cc: Add new config option use_doc_date to use document meta information for the DocTime() field. * htdig/HTML.cc(do_tag): Call Retriever::got_time if use_doc_date is set and we run across a META date tag. * htdig/Retriever.h, htdig/Retriver.cc: Add new got_date function. When called, sets the DocTime field of the DocumentRef after parsing is completed. Currently assumes ISO 8601 format for the date tag. Sun Apr 11 12:51:39 1999 Hans-Peter Nilsson * htsearch/Display.cc (buildMatchList): Delete thisRef if excluded by URL. Call setRef(NULL), not setRef(thisRef). Wed Apr 7 19:35:42 1999 Geoff Hutchison * htsearch/htsearch.cc(usage): Remove bogus -w flag. Thu Apr 1 12:05:11 1999 Gilles Detillieux * htsearch/htsearch.cc(main): Apply Gabriele's patch to avoid using an invalid matchesperpage CGI input variable. * htsearch/Display.cc(display) & (setVariables): Correct any invalid values for matches_per_page attribute to avoid div. by 0 error. Wed Mar 31 15:19:25 1999 Gilles Detillieux * htfuzzy/Synonym.cc: Fix previous fix of minor memory leak. (db pointer wasn't properly set) Mon Mar 29 10:31:09 1999 Geoff Hutchison * htsearch/Display.cc(excerpt): Added patch from Gabriele to improve display of excerpts--show top of description always, otherwise try to find the excerpt. Sun Mar 28 19:45:02 1999 Hans-Peter Nilsson * htlib/HtWordType.h (HtIsWordChar): Avoid matching 0 when using strchr. (HtIsStrictWordChar): Ditto. * htdig/ExternalParser.cc (parse): Before got_href call, set hopcount of URL to that of base plus 1. Add URL to external parser error output. * htlib/URL.cc (URL(char *ref, URL &parent) ): Move call to constructURL call inside previous else-clause. (parse): Reset _normal, _signature, _user initially. Commence parsing, even if no "//" is found. Do not set _normal here. (normalizePath): Call removeIndex finally. * htcommon/WordRecord.h (WORD_RECORD_COMPRESSED_FORMAT) [!NO_WORD_COUNT]: Change to "cu4". * htlib/HtPack.cc (htPack): Correct handling at end of code-string and end of encoding-byte. Add code 'c' for often-1 unsigned ints. (htUnpack): Add handling of code 'c'. Thu Mar 25 12:18:05 1999 Gilles Detillieux * installdir/long.html, installdir/short.html: Remove backslashes before quotes in HTML versions of the builtin templates. * Makefile.in: Add long.html & short.html to COMMONHTML list, so they get installed in common_dir. Thu Mar 25 11:56:50 1999 Gilles Detillieux * htsearch/Display.cc(displayMatch), htcommon/defaults.cc, htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Add date_format attribute suggested by Marc Pohl. Thu Mar 25 09:46:07 1999 Gilles Detillieux * htsearch/Display.cc(displayMatch): Avoid segfault when DocAnchors list has too few entries for current anchor number. Tue Mar 23 15:08:40 1999 Gilles Detillieux * htsearch/Display.cc(displayMatch): Fix problem when documents did not have descriptions. Tue Mar 23 14:17:14 1999 Gilles Detillieux * htdig/PDF.cc(parseString): Use minimum_word_length instead of hardcoded constant. Tue Mar 23 14:02:40 1999 Gilles Detillieux * htdig/HTML.cc: Fix bug where noindex_start was empty, allow case insensitive matching of noindex_start & noindex_end. * htdoc/attrs.html, htdoc/cf_byname.html, htdoc/cf_byprog.html: Fix inconsistencies in documentation for noindex_start & noindex_end. Tue Mar 23 14:01:16 1999 Gilles Detillieux * htdig/HTML.cc: Add check for tag that is missing a closing tag, terminating it at next href. Tue Mar 23 13:57:35 1999 Gilles Detillieux * htdig/Document.cc: Fix check of Content-type header in readHeader(), correcting bug introduced Jan 10 (for PR#91), and check against allowed external parsers. Tue Mar 23 13:54:35 1999 Gilles Detillieux * htdig/HTML.cc: More lenient comment parsing, allows extra dashes. Tue Mar 23 12:22:53 1999 Gilles Detillieux * htlib/Configuration.cc(Add): Fix function to avoid infinite loop on some systems, which don't allow all the letters in isalnum() that isalpha() does, e.g. accented ones. * htdig/HTML.cc: Fix three reported bugs about inconsistent handling of space and punctuation in title, href description & head. Now makes destinction between tags that cause word breaks and those that don't, and which of the latter add space. Tue Mar 23 12:15:48 1999 Gilles Detillieux * htdig/Plaintext.cc(parse): Use minimum_word_length instead of hardcoded constant. Tue Mar 23 12:11:04 1999 Gilles Detillieux * htmerge/words.cc(mergeWords): Fix to prevent description text words from clobbering anchor number of merged anchor text words. Tue Mar 23 12:02:00 1999 Gilles Detillieux * htsearch/Display.cc(generateStars): Add in support for use_star_image which was lost when template support was put in way back when. Tue Mar 23 11:47:52 1999 Gilles Detillieux * Makefile.in: add missing ';' in for loops, between fi & done Mon Mar 22 16:06:15 1999 Gilles Detillieux * htdig/HTML.cc: Check for presence of more than one tag. Mon Mar 22 15:32:15 1999 Gilles Detillieux <grdetil@scrc.umanitoba.ca> * contrib/parse_doc.pl: Fix handling of minimum word length. Sun Mar 21 15:19:00 1999 Hans-Peter Nilsson <hp@bitrange.com> * htlib/HtPack.cc (htPack): New. * htlib/HtPack.h: New. * htsearch/parser.cc (perform_push): Unpack WordRecords using htUnpack. * htsearch/htsearch.h: Add "debug" declaration. * htmerge/words.cc (mergeWords): Pack WordRecords using htPack. * htlib/Makefile.in (OBJS): Add HtPack.o * htcommon/WordRecord.h: Add WORD_RECORD_COMPRESSED_FORMAT * htdig/HTML.cc (parse): Keep contents in String variable textified_contents while using its "char *". * htsearch/Display.cc (excerpt): Similar for head_string. Thu Mar 18 20:01:24 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * installdir/long.html, installdir/short.html: Write out HTML versions of the builtin templates. * installdir/htdig.conf: Add commented-out template_map and template_name attributes to use the on-disk versions. Tue Mar 16 03:06:06 1999 Hans-Peter Nilsson <hp@bitrange.com> * htcommon/DocumentDB.cc (Delete): Fix bad parameter to Get: use key, not DocID. Tue Mar 16 01:50:16 1999 Hans-Peter Nilsson <hp@bitrange.com> * htlib/HtWordType.h (class HtWordType): New. * htlib/HtWordType.cc: New. * htlib/Makefile.in (OBJS): Add HtWordType.o * htdoc/attrs.html: Document attribute extra_word_characters. * htdoc/cf_byprog.html: Ditto. * htdoc/cf_byname.html: Ditto. * htcommon/defaults.cc (defaults): Add extra_word_characters. * htsearch/htsearch.h: Lose spurious extern declaration of unused variable valid_punctuation. * htsearch/htsearch.cc (main): Call HtWordType::Initialize. (setupWords): Use HtIsWordChar, HtIsStrictWordChar and HtStripPunctuation. Do not read valid_punctuation. * htsearch/Display.cc (excerpt): Use HtIsStrictWordChar. * htlib/StringMatch.cc (FindFirstWord): Ditto. (CompareWord): Ditto. * htdig/htdig.cc (main): Call HtWordType::Initialize. * htdig/Retriever.h (class Retriever): Lose member valid_punctuation. * htdig/Retriever.cc (Retriever): Lose its initialization. * htdig/Postscript.h (class Postscript): Lose member valid_punctuation. * htdig/Postscript.cc (Postscript): Lose its initialization. (flush_word): Use HtStripPunctuation. (parse_string): Use HtIsWordChar, HtIsStrictWordChar and HtStripPunctuation. * htdig/Parsable.h (class Parsable): Lose member valid_punctuation. * htdig/Parsable.cc (Parsable): Lose its initilization. * htcommon/WordList.cc (valid_word): Use HtIsStrictWordChar. (BadWordFile): Use HtStripPunctuation. Do not read valid_punctuation. * htcommon/DocumentRef.cc (AddDescription): Use HtIsWordChar, HtIsStrictWordChar and HtStripPunctuation. Do not read valid_punctuation. * htdig/PDF.cc (parseString): Similar.. * htdig/HTML.cc (parse): Similar. * htdig/Plaintext.cc (parse): Similar. Sun Mar 14 14:04:31 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/Makefile.in: Add HtSGMLEntites.o to OBJS. Sat Mar 13 21:29:38 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htcommon/DocumentDB.cc(Open, Read): Switch to DB_HASH for faster access. Most important for very quick URL lookups! * htcommon/DocumentRef.cc(AddDescription): Check to see that description isn't a null string or contains only whitespace before doing anything. * htlib/HtSGMLCodec.h, htlib/HtSGMLCodec.cc: Add new class to convert between SGML entities and high-bit characters. * htdig/HTML.cc(parse): Use it instead of SGMLEntities. * htsearch/Display.cc(excerpt): Use HtSGMLCodec to covert *back* to SGML entities before displaying. * htlib/HtHeap.cc: Cleaned up comments, use more efficient procedure to build from a vector. * htlib/HtWordCodec.cc(HtWordCodec): Fix bug with constructing from uninitialized variables! * htlib/URL.h, htlib/URL.cc: Initial support for multiple schemes and user@host URLs. * htlib/List.cc(Nth): Check for out-of-bounds requests before doing anything. Fri Mar 12 00:31:03 1999 Hans-Peter Nilsson <hp@bitrange.com> * htlib/mktime.c (__mon_yday): Correct size to number of initializers (2). * htsearch/htsearch.cc (main): Remove doc_index handling. * htsearch/ResultMatch.h (setURL): Change to setID, use int. All callers changed. (getURL): Change to getID. All callers changed. (String url): Change to "int id". * htsearch/Display.h: (Display): Second parameter removed. (docIndex) removed. * htsearch/Display.cc (Display, ~Display): Do not handle docIndex. (display): Use DocumentDB::operator [](int), not DocumentDB::operator [] (char *). (buildMatchList): Changed to handle ResultMatch as DocID int, instead of URL string: use DocumentDB::operator [](int), not DocumentDB::operator [] (char *). Get DocumentRef directly, then filter the URL by includeURL(). * htnotify/htnotify.cc (main): Use DocIDs(), not DocURLs(). Handle the change from String * to IntObject *. * htmerge/htmerge.cc (main): Do not delete doc_index. * htmerge/docs.cc (convertDocs): Test doc_index access as read-only. Pass as parameter for docdb, do not handle separately. * htmerge/docs.cc (convertDocs): Add debug messages about cause when deleting documents. If verbose > 1, write id/URL for every URL. * htmerge/db.cc (mergeDB): Handle doc_index, test accessibility. * htlib/IntObject.h (class IntObject): Add int-constructor. * htdoc/attrs.html (doc_index): Say that mapping is from document URLs to numbers. (doc_db): Say that indexing is on document number. * htdoc/cf_byprog.html (doc_index): Move from htsearch to htdig entry. * htdig/htdig.cc (main): Add .work suffix to doc_index too. Unlink doc_index if initial. * htcommon/DocumentDB.h (Open): New second argument. (Read): New second argument, default to 0. (operator [](int)): New. (Exists(char *), Delete(char *)): Change to int parameter. (DocIDs, i_dbf): New. * htcommon/DocumentDB.cc (operator [](int)): New. (Exists(char *), Delete(char *)): Changed to DocID int parameter. All callers changed. (URLs): Assume keys are ok without probing for documents with each key. (DocIDs): New. (Open): Take an index database file name as second argument. All callers changed. (Read): Similar, accept 0. (all): Change to index on DocID. Wed Mar 10 02:25:24 1999 Hans-Peter Nilsson <hp@bitrange.com> * htdoc/attrs.html (template_name): Typo; used by htsearch, not htdig. Mon Mar 8 13:30:44 1999 Hans-Peter Nilsson <hp@bitrange.com> * htdig/Retriever.cc (got_href): Check if the ref is for the current document before adding it to the db. Mon Mar 8 01:36:38 1999 Hans-Peter Nilsson <hp@bitrange.com> * htlib/DB2_db.cc: Remove errno. * htlib/DB2_hash.cc: Ditto. Sun Mar 7 20:50:37 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htfuzzy/EndingsDB.cc(createDB): Use link and unlink to move, rather than a non-portable system call. * htcommon/DocumentRef.h, htcommon/DocumentRef.cc: Fix #ifdef problems with zlib. Sun Mar 7 09:39:37 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/timegm.c: Fix problems compiling on libc5 systems noted by Hans-Peter. * htlib/Makefile.in, Makefile.in, Makefile.config.in: Use regex.c instead of rx. * htfuzzy/EndingsDB.cc: Ditto. * configure.in, configure: Don't bother to config rx directory. Fri Mar 5 08:09:20 1999 Gilles Detillieux <grdetil@scrc.umanitoba.ca> * contrig/parse_doc.pl: uses pdftotext to handle PDF files, generates a head record with punctuation intact, extra checks for file "wrappers" & check for MS Word signature (no longer defaults to catdoc), strip extra punct. from start & end of words, rehyphenate text from PDFs. Tue Mar 2 23:18:20 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htdig/htdig.cc: Renamed main.cc for consistency with other programs. * htlib/DB2_hash.h, htlib/DB2_hash.cc: Added interface to Berkeley hash database format. * htlib/Makefile.in: Use them! * htlib/Database.h: Define database types, allowing a choice between different formats. * htlib/Database.cc(getDatabaseInstance): Use passed type to pick between subclasses. Currently only uses Hash and B-Tree formats of Berkeley DB. * htcommon/DocumentDB.cc, htfuzzy/Endings.cc, htfuzzy/EndingsDB.cc, htfuzzy/Fuzzy.cc, htfuzzy/Prefix.cc, htfuzzy/Substring.cc, htfuzzy/Synonym.cc, htfuzzy/htfuzzy.cc, htmerge/docs.cc, htmerge/words.cc, htsearch/Display.cc, htsearch/htsearch.cc: Use new form of getDatabaseInstance(), currently with DB_BTREE option (for compatibility). Mon Mar 1 22:53:37 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/regex.c, htlib/striptime.c: Import new versions from glibc. * htlib/Makefile.in, htlib/mktime.c, htlib/timegm.c, htlib/lib.h: Changes to use glibc timegm() function instead of buggy mytimegm(). * htdig/Document.cc(getdate): Use it. Tue Mar 2 02:35:50 1999 Hans-Peter Nilsson <hp@bitrange.com> * attrs.html: Rephrase and clarify entry for url_part_aliases. Sun Feb 28 23:25:40 1999 Hans-Peter Nilsson <hp@bitrange.com> * htlib/HtURLCodec.cc (~HtURLCodec): Add missing deletion of myWordCodec. Fri Feb 26 19:03:58 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * configure, configure.in: Fix typo on timegm test. * htlib/mytimegm.cc: Fix Y2K problems. Wed Feb 24 21:09:19 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htsearch/htsearch.cc(main): Remember to delete the parser! * htlib/String.cc(String(char *s, int len)): Remove redundant copy. * htsearch/Display.cc(display): Free DocumentRef memory after displaying them. (displayMatch): Fix memory leak when documents did not have anchors. Wed Feb 24 15:18:26 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/Configuration.cc(Add): Fix small leak in locale code. * htlib/String.cc: Fix up code to be cleaner with memory allocation, inline next_power_of_2. Mon Feb 22 22:13:49 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/String.cc, htlib/htString.h: Fix some memory leaks. Mon Feb 22 08:52:19 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/Dictionary.h, htlib/Dictionary.cc(hashCode): Check if key can be converted to an integer using strtol. If so, use the integer as the hash code. * htlib/HtVector.h, htlib/HtVector.cc: Implement Release() method and make sure delete calls are done properly. * htsearch/ResultList.h, htsearch/ResultList.cc(elements): Use HtVector instead of List. * htsearch/parser.cc: Ditto. Sun Feb 21 16:13:59 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/HtHeap.h, htlib/HtHeap.cc: Add new class. * htlib/Makefile.in: Compile it. * htlib/HtVector.h, htlib/HtVector.cc: Add Assign() to assign to elements of vectors. Sun Feb 21 14:45:26 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htsearch/htsearch.cc: Add patch from Jerome Alet <alet@unice.fr> to allow '.' in config field but NOT './' for security reasons. * htdig/HTML.cc: Add patch from Gabriele to ensure META descriptions are parsed, even if 'description' is added to the keyword list. Sun Feb 21 14:43:44 1999 Gilles Detillieux <grdetil@scrc.umanitoba.ca> * htsearch/parser.h, htsearch/parser.cc: Clean up patch made for error messages, made on Feb 16. Thu Feb 18 20:19:30 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * htlib/HtVector.h, htlib/HtVector.cc: Added new Vector class. * htlib/Makefile.in: Compile it. * htlib/strptime.c: Add new version from glibc-2.1, replacing strptime.cc. * htdig/Document.cc: Use it. * htlib/regex.h, htlib/regex.c: Add new files from glibc-2.1. * htlib/mktime.c: Update from glibc-2.1. Wed Feb 17 23:44:59 1999 Geoff Hutchison <ghutchis@wso.williams.edu> * configure.in, configure, aclocal.m4: Add autoconf macro to detect syntax of makefile includes. * Makefile.in, Makefile.config.in, */Makefile.in: Change include syntax to use it. Wed Feb 17 12:36:42 1999 Hans-Peter Nilsson <hp@bitrange.com> * htcommon/defaults.cc (defaults): locale: change to "C".