Creating your own cvsupd server

Creating your own cvsupd server

With the upcoming work I’m going to be doing on FreshPorts, I thought it best to start up a cvsup server
for the source code.  This server will be used to share the data amongst the other
FreshPorts developers.  This article starts that process.

I first wrote about cvsupd
on 6 August 1999.  That article is still around, but now
that I’m installing a cvsupd server for my own needs, instead of creating a mirror, I found my existing notes a bit lacking. 
Hence, this article.

I used John Polstra’s CVSUP
FAQ for this exercise.  Of note is the section on setting up a test server.
  It’s what I used as the basis for this article.

The install

Well, seeing as I had all the port skeletons installed, I
just had to do the following:


cd /usr/ports/net/cvsupd-bin
make install


NOTE: in more recent versions of FreeBSD, cvsupd can be found in net/cvsup-without-gui.

Yet cvsupd is started by /usr/local/etc/rc.d/cvsupd which is installed by net/cvsup-mirror

cvsupd_enable=”YES” in /etc/rc.conf

cvsupd_flags=”-c sup:/home/repositories/sup”

Turn off your old cvsupd to avoid accidents.

Setting up a test server

In this section, I’m going to show you the bare minimum required to get
the server running so you can talk to it from a client.

• Create the cvsupd base directory:

This is the main directory which cvsupd uses and its where the repositories
will reside.  In this example, we’ll use /usr/local/etc/cvsup (which is
the default base directory for cvsupd).  See the -b option.

• Create the collection directory:

(i.e. mkdir /usr/local/etc/cvsupd/sup).  See the -c option.

• Create the test collection directory:

This directory will contain the test collection
(i.e. mkdir /usr/local/etc/cvsupd/sup/test)

• Create the releases file:

The releases file identifies the releases which are associated with this
collection.
Create /usr/local/etc/cvsupd/sup/test/releases to contain the following:


cvs list=list.cvs prefix=/home/repositories


This defines the prefix.  Our repository will exist at /home/repositories

• Create the list.cvs file:

This file contains the rules for use when processing the cvs release.  In this
case, it contains the freshports-db collection.  Here’s what’s in that
file:

upgrade freshports-db

This file should be in the directory /usr/local/etc/cvsupd/sup/test/.

This is your repository and it is relative to the prefix supplied in the releases
file.  In our example, the repository will exist at /home/repositories/freshports-db.
  It is the contents of this subtree which will be transferred.

Start the server

Here’s where we start the server with a very simple situation:

/usr/local/sbin/cvsupd

We have not specified a base directory.  cvsupd will use the default
directory, which we also used to create our base directory.  We
have not specified the -e option, which means cvsupd will run from the command
line and not go into the background.  Here is what it looks like when cvsupd
starts:

# /usr/local/sbin/cvsupd
2000.12.17 16:54:40 NZDT [75872]: CVSup server started
2000.12.17 16:54:40 NZDT [75872]: Software version: REL_16_1
2000.12.17 16:54:40 NZDT [75872]: Protocol version: 16.1
2000.12.17 16:54:40 NZDT [75872]: Ready to service requests


This method is fine for testing, but I actually run cvsupd as the nobody and
ensure that everything it serves is readable by everyone.  cvsupd does not
create or write files, so from a security point of view, the risk of cvsupd being
tricked into damaging your system is actually quite low.

I start cvsupd automatically at system startup by creating /usr/local/etc/rc.d/cvsupd.sh,
which is chmod 770:

#!/bin/sh
[ -x /usr/local/sbin/cvsupd ] && \
su -m nobody -c "/usr/local/sbin/cvsupd -e -C 8 -l @daemon" && \
echo -n ' cvsupd'


Setting up a test client

We will now set up the test client to use our test server.  We are
doing this on the same box as cvsupd is running. Create the directory ~/cvs-test. 
In this directory, create supfile and place the following
in it:

*default host=localhost
*default base=.
*default release=cvs
*default delete use-rel-suffix
test


This is our cvsup configuration file.  We have specified that we will be
connecting to the cvsup server on localhost (i.e. this box) and that we
will be obtaining the cvs release.  From that release, we want the test
collection.

To run cvsup on the client and pull down the collection, issue the following command:

cvsup supfile

Here is a successful connection and cvsup:


$ cvsup supfile
Connected to localhost
Updating collection test/cvs
 Mkdir freshports-db
 Create freshports-db/freshports.db.sql,v
 Create freshports-db/permissions.txt,v
 SetAttrs freshports-db
Finished successfully


In this case, it’s brought down the files associated with my repository.

Another example: test2

In this example, I’m going to add a very simple collection, test2,
which contains only contrib/top.
  This isn’t really a practical example, but it might help to illustrate how the
various components of cvsup fit together.

I’ll create the collection directory:

mkdir /usr/local/etc/cvsup/sup/test2

In that directory, I’ll specify the releases associated with this collection by
creating releases which contains the following (all on one line):

cvs list=list.cvs prefix=/home/repositories/FreeBSD/ncvs/src/contrib/

This indicates that the repository actually exists at /home/repositories/FreeBSD/ncvs/src/contrib/.

Then I create the list.cvs file, the same directory as releases, to
contain this:

upgrade test2

I can use the same supfile as in the previous example, but I change test
to top.

Now when I cvsup, I get this:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout top/ADVERTISEMENT
Checkout top/Changes
Checkout top/Configure
Checkout top/DISCLAIMER
Checkout top/FAQ
Checkout top/INSTALL
Checkout top/Make.desc.X
Checkout top/Makefile.X
Checkout top/Porting
Checkout top/README
Checkout top/boolean.h
Checkout top/commands.c
Checkout top/display.c
Checkout top/display.h
Checkout top/getans
Checkout top/getopt.c
Checkout top/install
Checkout top/layout.h
Checkout top/loadavg.h
Checkout top/m-template
Checkout top/machine.h
Checkout top/metatop
Checkout top/os.h
Checkout top/patchlevel.h
Checkout top/prime.c
Checkout top/screen.c
Checkout top/screen.h
Checkout top/sigconv.awk
Checkout top/top.X
Checkout top/top.c
Checkout top/top.h
Checkout top/top.local.H
Checkout top/username.c
Checkout top/utils.c
Checkout top/utils.h
Checkout top/version.c
Finished successfully

Compare the above list of files you can find in the CVS repository for contrib/top.

And if I check my directory, I see this:

$ ls
sup supfile top

And if I look in top, I see the same files as are listed above.

Adding another bit to test2

Now lets try another example.  Let’s add contrib/awk to
the test2 collection.

I’ll assumed you have already created the example
from the previous section.  To add awk to the
collection, modify /usr/local/etc/cvsup/sup/test2/list.cvs to contain this:

upgrade top
upgrade awk

Now run cvsup again.  You should see this:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout awk/ACKNOWLEDGMENT
Checkout awk/COPYING
Checkout awk/ChangeLog

[snip]

Checkout awk/test/subslash.awk
Checkout awk/test/subslash.ok
Checkout awk/test/zeroflag.awk
Checkout awk/test/zeroflag.ok
Checkout awk/version.c
Finished successfully

and your main directory should contain this:

$ ls
awk sup supfile top

Putting things in a different place

The supfile example contains a base specifiation of base=.
which puts everything from cvsup into the current directory.  That’s not the ideal
situation, but it’s what we used for our testing.  Now let’s change that.  Let’s
put everything into /home/dan/mysupfiles.  The first step is to modify
our supfile from the previous examples and change base to look like this:

*default base=/home/dan/mysupfiles

Don’t forget to create the directory before you run cvsup:

mkdir /home/dan/mysupfiles

Then we cvsup again:

$ cvsup supfile
Connected to localhost
Updating collection test2/cvs
Checkout awk/ACKNOWLEDGMENT
Checkout awk/COPYING
Checkout awk/ChangeLog
[snip]
Checkout top/top.h
Checkout top/top.local.H
Checkout top/username.c
Checkout top/utils.c
Checkout top/utils.h
Checkout top/version.c
Finished successfully

Now if I check my directory, I’ll find this:

$ ls /home/dan/mysupfiles/
awk sup top

NOTE: If you get this error:

$ cvsup supfile
Nonexistent base directory "~/mysupfiles" for collection "test2"

Then you didn’t create ~/mysupfiles before running cvsup.

Authenticating access to your cvsup server

This section will concentrate on restricting access to your cvsup server
by authenticating the connections.  Authentication is controlled by the existence of
the file cvsupd.access which must appear in the base directory (in the example
above, the base directory is /usr/local/etc/cvsup).  If this file does not
exist, no authentication takes place.

The file cvsupd.passwd (again, in the
base directory) contains the authentication rules.  Here’s the contents I was testing
with:

cvsup.example.org:mysecretkey
dan@develop.example.org:$md5$c2e5a85c8042ce9f8c71bcc0f52876c8::


mysecretkey can be any key you choose.  The second line was created using cvpasswd:

$ cvpasswd dan@develop.example.org cvsup.example.org
Enter password:
Enter same password again:

Send this line to the server administrator at cvsup.example.org:
----------------------------------------------------------------
dan@develop.example.org:$md5$c2e5a85c8042ce9f8c71bcc0f52876c8::
----------------------------------------------------------------
Be sure to send it using a secure channel!

Add this line to your file "$HOME/.cvsup/auth", replacing "XXX"
with the password you typed in:
----------------------------------------------------------------
cvsup.example.org:dan@develop.example.org:XXX:
----------------------------------------------------------------
Make sure the file is readable and writable only by you!


The password I entered was "secret".  So this is what I placed in ~/.cvsup/auth
on my client machine:

cvsup.example.org:dan@develop.example.org:secret:

That should be enough to authenticate everyone.

Access Control

Perhaps you don’t want everyone accessing your cvsup server. I know I don’t. So I’ve added some
rules to my packet filter to restrict access. People scanning the box can’t
even tell there is a cvsup server there. But multiple layers of protection can be a good thing. That’s
why I added more. If you read the man page for cvsupd, you’ll see that that cvsupd.access
is your friend. I added a few rules to this file, which normally resides under
/usr/local/etc/cvsup, which permitted certain trusted host unlimited access to my server.


#
# allow anything from our local network

+10.0.0.0/24

# allow connections from local host
+127.0.0.1/32

# everything else must authenticate
*0.0.0.0/0


Using the above scenario, I didn’t have to do anything special for boxes on my 10.0.0.* network
and the cvsup server could access itself. Everyone else, including my remote boxes, must
authenticate using the methods described in the previous section.

Secrecy

From man cvsupd:

If secrecy is desired then the connection can be tunneled
through ssh.


I tried to get this to work, but I failed.  If anyone has a practical example,
please add it to the comments section. Thanks.

#!/bin/sh
ssh -f -L 32884:example.org:5999 dan@example.org sleep 30
cvsup -h localhost -p 32884 supfile


Common problems and solutions

If you encounter this message:

$ cvsup supfile
Connected to localhost
Server message: Collection "test" release "cvs" is not available here
Skipping collection test/cvs
Finished successfully


…it means the information contained in the releases file is
incorrect.  In my case, I had put my data at /usr/repositories instead of /home/repositories
as specified in releases.  Once I moved repositories to /home,
all was well.

If you encounter this message:


$ cvsup supfile
Connected to localhost
Server message: Unknown collection "test"
Skipping collection test/cvs
Finished successfully


…it means the collection specified in your client’s supfile does not exist on the
server.  Once of them is wrong.  In our example, it means /usr/local/etc/cvsupd/sup/test
does not exist.

This message:

$ cvsup supfile
Cannot connect to localhost: Connection refused
Will retry at 16:53:45


…means that cvsupd isn’t accepting connections.  Make sure it’s
running.  Make sure you are allowed to connect.  Check your authentication.

If you get this:

$ cvsup supfile
Connected to cvsup.freshports.org
Server error: Authentication failed


…then you have the wrong authentication information at either the client or the
server.  Check both and try again.