udmSearch - News Extension README
Author: Heiko Stoermer <firstname.lastname@example.org>
Since 3.10 udmSearch comes with an integrated extension to archive news
servers. (currently MySQL only! see restrictions)
This means that you can now download all messages from a news server
an save them completely in a database.
you can expire the messages on the news server to keep it slim and fast
you can search the complete message base with all the features that regular
you can still browse discussion threads over the complete archive
currently mysql only (I would have really liked to do this for postgresql,
but some really annoying restrictions concerning query size and field size
in postgresql finally made me switch to mysql.)
single dict only (because mysql-perl frontend does not support multi-dict)
No new features are planned for this thing. It works the way it is (at
least as far as I can see) and does everything I wanted it to do.
What I will do is to make the code a bit more portable to other databases
and fix the few very tiny bugs in the frontend.
Of course newly discovered bugs will be fixed. I'm maintaining it as
good as I can.
Of course, important questions always are: how fast.../how big.../how long....
Our local intranet installation of udmSearch says the following:
76132 76132 OK
119 119 Not found
17 17 Service Unavailable
802 802 Gateway Timeout
which means that roughly 77.000 messages are archived in the
Current database size is: 423 Megabytes
The dict table has 6.076.462 entries
It's run on an AMD K6 400 with 64 MBs of RAM (very tiny thing)
typical queries take between 2 and 10 seconds.
Unpack the udmSearch distribution archive.
start the configure script with the options --enable-news-extension
make and make install as described in the regular install instructions
The news extension uses a slightly different database layout. The create
files can be found in frontends/mysql-perl-news/create/
(Of course you have to do mysqladmin create udmsearch first
and set permissions to the account the web-frontend and indexer are run
an indexer.conf for incremental news archiving (messages hardly ever change...)
can be found in frontends/mysql-perl-news/etc/ together with a sample
cron shell script that can be run once a day or so.
Please see indexer.conf for detailed description of the indexing
Install perl frontend:
copy frontends/mysql-perl-news/*.pl and frontends/mysql-perl-news/*.htm*
to your cgi-bin directory.
copy frontends/mysql-perl-news/*.pm to your site's perl library
dir (site_perl or so) where the modules can be found by the perl scripts.
edit search.htm and change the included database login information
The Perl frontend has additional features that allow you to browse
message threads. You will see.
Now you are set and can run indexer for the first time according to the
instructions you can find in indexer.conf.
I hope this is a nice feature for you.
If anyone is interested in porting this to other databases/multidict
mode/the PHP frontend, PLEASE DO SO! I would be pleased and will