Migrating a news spool...

[Dave Laird]

Good afternoon, everyone...

PART III. Migrating a news spool...

One of the recurring problems of running a news server for a prolonged
period of time is eventually you have to deal with the news spool,
containing all the messages you have accumulated over the last decade.
Some are merely in-transit, waiting to be expired normally during daily
maintenance, and as such can be tossed unless they have a specific
purpose. Others, such as the used_kharma.* and spk.* hierarchies, which
are the archives of the last ten-plus years, need to be migrated to
whatever the new news server may become.

This is far from being a new problem. Fortunately, the last time I moved
the spool to new drives, I changed the storage format of the articles
themselves to tradspool, which means that the entire message base consists
of individual files. In the case of used_kharma.chat, that encompasses
nearly 2000 individual files, all of which need to move to another news
spool and be imported.

Migrating the news spool has always been one of the biggest problems that
news admins face. Traditional logic has it that you can perform this
operation using innxmit and some judicious scripting. In several articles
written by Rich Salz, the author of INND and its supporting applications,
the overt statement has been made that you can use innxmit to migrate an
existing news spool from one server to another.

While I have continued testing various flavors of Linux, I have been also
performing a series of ongoing experiments with news spool migration with
an eye toward resolving this problem in the near future.

My first effort, using Rich Salz's own methods, consisted of feeding the
entire Kharma Lot news spool to innxmit and from there, over the network
to the new news server, in this case running Debian INN2. Setting up the
file containing the list of messages to be migrated didn't take but six
minutes. However, once I fired off innxmit I immediately noticed that it
effectively shut down Kharma's own news server, because the load on the
processor shot through the roof and stayed there. According to my figures,
it was going to take about 1 hour, forty-six minutes, minutes, give or
take a few, to complete the migration. The minute I saw that the news
servers on both ends were essentially crippled by the burden of using
innxmit, I knew there had to be a better method, especially considering
how difficult it was to set up the file list for innxmit.

God forbid that I should ever possess a truly huge news spool! I can only
imagine the number of weeks it would take, for example, for a
multi-terabyte news spool to migrate, using this method. From discussions
with other news admins, I am aware they have ultra high-speed tape drives
for transiting news spools from one machine to another. The average price
of one of these beauties runs about a thousand dollars, or though your
results may differ.

My second effort, which completed late last night, while much simpler,
using a Magnum tape drive and three tapes, still has problems of its own.
The unfortunate part is that using a tape drive requires that someone
physically sit and feed the tapes into the drive. Despite the Magnum being
a pretty high-speed SCSI device, it has always taken about forty-five
minutes to back up the entire news spool. Fortunately, the tape backup
process does not adversely affect the news server or its spool.

My latest effort, which was concluded this afternoon using a different
technology entirely, is interesting in that it requires neither personal
intervention nor does it impact the news server's operation.

With just a minor amount of juggling the permissions on the directory tree
on the *incoming* news server, and by allowing the super-user to connect
to the server over SSH, I have been able to copy the entire news spool in
under an hour, using the SCP utility built into the Secure Shell
application. The beauty of this method is that it takes roughly half the
time of a tape backup, can be scripted, and requires only that the history
file(s) be re-generated on the new news server, which one would ordinarily
do anyway.

This, of course, requires that each respective news server be running a
news spool in the traditional tradspool format. I suppose, if this series
of articles were to be consistent and 100% accurate, I probably should try
performing a migration where the two news spools were dissimilar enough to
require the use of innxit, but that does not seem to fit into my time
commitment today.

Comments and opinions, of course, are appreciated.

Dave

[Dave Laird]

Good afternoon, James...

Try newsx.

Ironically, that might work, but it requires X. However, if I don't find
another method, I might load X long enough to migrate the news spool.
Thanks, though. I had all but forgotten newsx in the testing phase.

Dave

[James Vahn]

Post by Dave Laird

Try newsx.

Ironically, that might work, but it requires X. However, if I don't find
another method, I might load X long enough to migrate the news spool.
Thanks, though. I had all but forgotten newsx in the testing phase.

newsx is "an NNTP client for posting and fetching news", generally
running from cron or scripted.

Install it on the Debian box and create the newsgroup (just one for this
test). In /etc/news/newsfeeds put "news.kharma.net:*:Tf:" and then reload.

As user "news" on the Debian box, run this from a command line:
newsx -d -W 0 --max-path 999 --no-path --inn --rnews news.kharma.net

You might have to specify the full path: /usr/lib/news/bin/newsx

Ctrl-c will stop it, but it stands a chance of working as-is. It should just
get a feed on whatever matches the active list and the "*" wildcard in the
newsfeed entry. The -W0" tells it there's no hurry.

If it works, "ctlinnd newgroup another.one y" to create another entry in the
active list, and run newsx again. With any luck at all, you can just create
the entries all at once, running newsx just one time.


--

[James Vahn]

Post by Dave Laird
Good afternoon, everyone...
PART III. Migrating a news spool...

Try newsx.

nnrpd(8) has support for controlling high-volume posters via an expo-
nential backoff algorithm, as configured by the following parameters.
Exponential posting backoff works as follows: News clients are indexed
by IP address (or username, see backoffauth below). Each time a post
is received from an IP address, the time of posting is stored (along
with the previous sleep time, see below). After a configurable number
of posts in a configurable period of time, nnrpd(8) will activate post-
ing backoff and begin to sleep for increasing periods of time before
actually posting anything. Posts will still be accepted, but at an
increasingly reduced rate.


--