NetSaint – a network monitor

 NetSaint, Network monitoring  Add comments
Sep 102001
 

NetSaint – a network monitor

NetSaint is a network monitor. You can use it to keep close tabs on your routers, printers, computers, and services. NetSaint can watch/monitor various services (SMTP, POP3, HTTP) to make sure they keep running. It can ping boxes to make sure they are still up. If a problem occurs, NetSaint can send out email, page people, and escalate problems if they persist. When the problem is fixed, NetSaint can also send out a recovery notification. And my personal favourite, you can view the current status through a webpage.

Regular readers will know that in early 2000, I wrote about another network monitor, Big Brother.

NetSaint uses a feature called plugins to perform the actual service checks. This is a very flexible approach which allows new plugins to be created for specific situations as and when necessary. If also allows you to create your own plugins if you so wish. In fact, there is a port for the plugins (net/netsaint-plugins). This port is a dependency of NetSaint and will be installed when you install NetSaint.

Installing NetSaint

It seems like everything I install is already available in the ports collection. That makes life simple. Here’s what you do: 
cd /usr/ports/net/netsaint
make install
Don’t forget to do a make clean afterwards. Otherwise all the files created by the build will sit around clutting up your disk. The NetSaint port has a fair number of dependencies (including gd, bzip2, etc, about a dozen in total) so it may take some time to install everything. But that’s the beauty of the ports system. It’s just start and forget.

But don’t forget it for too long. You will eventually be given a list of things to choose from:

NetSaint plugins configuration options

 Please select desired options:

    [ ] QStat         Game server query support
    [X] FPing         Support for non-flooding fast ping
    [X] NetSNMP       SNMP support
    [ ] Radiusclient  Radius support
    [X] MySQL         MySQL support
    [X] PostgreSQL    PostgreSQL support
    [ ] OpenLDAP      OpenLDAP support

            [  OK  ]       Cancel
You can see the selections I choose. Only you can figure out what makes sense to you. If none of the above options make any sense, don’t select them. If you’re not running mysql or postgresql, then don’t worry about those options.

I answered y to both of the following questions:

You need a group "netsaint".
Would you like me to create it [y]?
Done.
You need a user "netsaint".
Would you like me to create it [y]?
NetSaint will run as a specific user, which is a good thing from a security point of view. The use of a specific group is also a good security concept. Combining the use of a specific user and a specific group for NetSaint is a basic and commonly used security precaution. Everything, including your NetSaint configuration files should be treated on a ‘need to know’ basis.

The web interface

This section may seem out of place, but I think it’s a good idea to get the web interface set up and at least partially functional before we go very far down the track. It also has the added benefit of giving you online access to the documentation….

You need to add the following to your apache configuration file (normally, this is /usr/local/etc/apache/httpd.conf). You should place these directives in the vhost section for the website you wish to use for NetSaint.
Alias /netsaint /usr/local/share/netsaint
ScriptAlias /cgi-bin/netsaint/ /usr/local/share/netsaint/sbin/
For a complete working example, see samples/netsaint.vhosts.txt.

Take note that the Alias does not contain a trailing /. On this point, I disagree with the NetSaint documentation where it specifies a trailing slash. I originally followed the documentation but encountered a “The page cannot be found” message when I did not specify a trailing slash in my browser. Removing the trailing slash from the Apache configuration file solved that problem.

To further restrict access to NetSaint, I also added the following to my Apache configuration file for the vhost in question:

<Directory "/usr/local/share/netsaint/sbin">
       AllowOverride None AuthConfig
       Options FollowSymLinks ExecCGI
       Order allow,deny
       Allow from 192.168.0.99
</Directory>

The AuthConfig will be used later to authenticate people using NetSaint. The ‘Allow from’ directive will restrict access to the CGI directory to the specified IP address. You can specify multiple IP addresses or hostnames on that line, separated by commas.

If you are even more worried about access, try adding this entry:

<Directory "/usr/local/share/netsaint">
        Order deny,allow
        Deny from all
        Allow from 192.168.0.99
</Directory>

This will prevent anyone from even getting to /netsaint/ unless they are at that IP.

You should then test your Apache changes:

# /usr/local/sbin/apachectl configtest

You should then HUP Apache to invoke your changes:

# /usr/local/sbin/apachectl graceful

You should then be able to browse to your NetSaint directory using http://www.example.org/netsaint/ (subsitute your own domain). But if you click on any of the CGI links, you may see an error something like this:

Error: Could not read host and service status information!

That’s because NetSaint isn’t running yet with host/service information. But we will deal with that problem after we complete the configuration.

Seen this error? (added on 27 January 2003)

If you encounter this error:
Error: Could not open CGI configuration file '/usr/local/etc/netsaint/nscgi.cfg' for reading!
If so, then you haven’t done what the next section tells you to do!

Configuration overview

NetSaint configuration files are located at /usr/local/etc/netsaint. The main configuration files are:

netsaint.cfg
The main configuration file for NetSaint. Basically, I didn’t do anything to this file.
hosts.cfg
The hosts file is where you define what you want NetSaint to monitor. It allows you to specify the computers, collect them into groups for ease of use, and specify who should be notified for a given problem. It was this file which took up most of my configuration time. The hosts.cfg can be split up into almost as many sub config files as you want which is especially useful to keep track of monitoring a large network.
commands.cfg
The commands are what NetSaint uses to perform various tasks. I modified this file slightly, but nothing major.
nscgi.cfg
NetSaint provides a web interface and makes heavy use of CGI programs. Access to these CGI programs is controlled by this file.
After the first install, you need to copy the distributed files to the file names expected by NetSaint: 
cp commands.cfg.dist commands.cfg
cp netsaint.cfg.dist netsaint.cfg
cp nscgi.cfg.dist    nscgi.cfg
cp resource.cfg.dist resource.cfg
chown netsaint *
chmod o-r *

WARNING: I just removed world-readability using the above step. I run Apache as www. So I used this to allow Apache to read the above (note: this tip may apply to other items as well, which I may have neglected to include; you’ll soon find out….):

chgrp www *

I purposely have not taken the distributed hosts file (hosts.cfg.dist) as I’m going to provide a simple example later in the next section.

That chown ensures the files are owned by netsaint rather than root. And the chmod makes sure nobody but netsaint can read the files. Both of these steps may be considered overkill, but I think it’s a good security measure. Keep things tidy.

hosts.cfg

The distributed hosts.cfg contains a very rich example. I suggest that you closely examine that file and use that as the basis for your actual configuration. But first, as a practical example, I suggest you use the following example and comment out the existing sections. This will get you going and allow you to see a small example with a minimum of fuss. Start small. Walk first. Then run. The sample hosts file is provided via this link. I don’t want to display it in this article because some of the lines are very long. You will need to make modifications to this sample file. Be sure to get the IP address right. Fill in the name, probably your own name. That name will be used in the next section for authentication.

nscgi.cfg

nscgi.cfg controls access to the CGI programs. It’s pretty obvious that you don’t want just anyone to see the system monitor information. The official information for this step is contained in your online documentation (available via the /netsaint/docs/cgiauth.html link) or at NetSaint. The short version of the story is we use .htaccess and a database in order to authenticate users via CGI. For more information about .htaccess files, try reading Protected Apache directories. In the meantime, I’ll just press ahead with a simple setup. The directory you want to protect is /usr/local/share/netsaint/sbin. You can protect it by placing a .htaccess file in that directory. In that file you would place the following: 
AuthName "NetSaint Access"
AuthType Basic
AuthUserFile /usr/local/etc/netsaint/htpasswd.users
require valid-user
You then create a password entry using the steps shown in Protected Apache directories. Be sure to use the same name as you used in the hosts.cfg file. You add this entry to /usr/local/etc/netsaint/htpasswd.users: 
dan:esaBHd2tW29g2
After doing those changes, Apache will prompt you for a user name and password when you broswe to one of the CGI directories (e.g. Status Overview).

The next step is to specify what commands are avialble to you. This is done via the nscgi.cfg. To get you started, just add the following. WARNING: This may be more than you want to allow. Read the documentation.

authorized_for_all_hosts=dan
authorized_for_all_host_commands=dan
authorized_for_all_services=dan
authorized_for_system_information=dan
authorized_for_system_commands=dan
authorized_for_configuration_information=dan

Now it’s time to start NetSaint and see what happens….

Starting NetSaint

Starting NetSaint is easy:

# /usr/local/etc/rc.d/netsaint.sh start
Starting network monitor: netsaint
PID TT STAT TIME COMMAND
88260 ?? Ss 0:00.03 /usr/local/bin/netsaint -d /usr/local/etc/netsaint/netsaint.cfg

Similarly, you can use the stop and restart options.

NetSaint process may not be running!

If you happen to see the message ‘NetSaint process may not be running!’ when you broswe to /netsaint, don’t worry. First, verify that NetSaint is indeed running. You should see something like this:
$ ps auwx | grep netsaint
netsaint 89648 0.0 2.1 1876 1288 ?? Ss 7:54PM 0:00.11 /usr/local/bin/netsaint -d /usr/local/etc/netsaint/netsaint.cfg

If you see something like that, then you know that NetSatint is actually running. If you click on the link ‘Click here for more info.’ (right below ‘NetSaint process may not be running!’), you’ll find that you need to modify nscgi.cfg and supply a value for netsaint_check_command. That’s pretty easy. All you need to do is uncomment the line on which that keyword is found and restart NetSaint.

OK, you should be running now

You should have NetSaint running now. There’s no reason why it shouldn’t be running. If you are having trouble, and it’s because of something not covered in this article, please add your solution/problem to the article comments. In any case, please consider this. Thanks.

Optional configuration

The steps in this section are optional. They may not be applicable to you. I strongly urge you to get everything else working, then consider these steps. In this section, I’m going to restrict things. I’m not going to let just anyone read my NetSaint configuration files. I’m only going to let the netsaint user read them. Let’s just say a little bit of paranoia is a good thing when it comes to security.

The first step is to chmod o-r the configuration files.

cd /usr/local/etc/netsaint
chmod o-r *

But if you do just that, you’ll find that Apache can’t read htpasswd.users. This will manifest itself as authentation failures in your Apache logs:

[error] [client 24.102.95.54] user dan not found: /cgi-bin/netsaint/status.cgi

The solution is to make htpasswd.users readable by Apache:

chgrp www htpasswd.users

But then you’ll have trouble with the other configuration files as well. So I found it easiest to just do this:

chgrp www *
That will go some way to restricting access to your configuration files but still allow the web interface to function.

FreeBSD graphics for NetSaint

NOTE: The graphics rerferred to by in this section are now included in the FreeBSD port. The steps below are no longer necessary. If you have a look at the web pages for NetSaint, you’ll notice it’s rather Linux-centric. Fair enough. It is designed for Linux (it says so right on their homepage. But RevDigger decided that some better graphics were in order. In the spirit of sharing, he’s allowed me to distribute these graphics. The graphics are available at: These files should be place at /usr/local/share/netsaint/images There are only two simple changes you need to make. Remember to change both the URL and the graphic dimensions. I won’t provide a diff. Merely a simple before-and-after. If the graphics don’t look right, you probably messed up on the dimensions. The filee you want to change are located at /usr/local/share/netsaint/.
  • side.html
    <img src="/images/sblogo.gif"     width="130" height="125" border="0">
    ---
    <img src="/images/bsd_sblogo.gif" width="110" height="95"  border="0">
  • main.html
    <IMG SRC="/images/logofullsize.gif"     BORDER="0" ALT="NetSaint" width="238" height="227">
    ---
    <IMG SRC="/images/bsd_logofullsize.gif" BORDER="0" ALT="NetSaint" width="268" height="221">

Possible problems

I think NetSaint has the ability to change configuration items from the webpage interface. If so, you may need to make the configuration files chmod +w with respect to your http process. In my case, I’m running Apache as www. So I made the NetSaint configuration files (/usr/local/etc/netsaint) a member of the www group and chmod g+r.

If you happen to see the following message:

It appears as though you do not have permission to view information for any of the hosts you requested…

Then I suspect your .htaccess user name and your NetSaint user names are not matching up. Or you didn’t specify any permission for your user name in nscgi.cfg.

If you keep seeing messages like this on your screen:

/home/dan/.mail_aliases: Permission denied
/home/dan/.mail_aliases: Permission denied

Or these in /var/log/messages:

forward /usr/local/var/netsaint/.forward.dev+: Group writable directory
Then I know of a change which will fix the problem (if it hasn’t already made it into NetSaint>. Modify /usr/local/etc/rc.d/netsaint.sh to add a single line (unset -v HOME) as shown below: 
rm -f $NetsaintCmd
unset -v HOME
$NetsaintBin -d $NetsaintCfg

That should stop the messages.

Thanks

I wish to thank the following people for reviewing this article.
  • Shane Allen
  • Atle Veka
It’s always helpful to have people read what you’ve written before it is released upon the world.

Was that useful

If you found that article useful, then please consider supporting this website. At the moment, we’re trying to raise funds to pay for the new year of colocation. Thank you.

  19 Responses to “NetSaint – a network monitor”

  1. Brandon D. Valentine says:
    September 11, 2001 at 12:11 am

    Those BSD graphics are pretty sharp. Someone really oughta write a patch and add that to the port so that on FreeBSD netsaint installs with those by default. Hmm…checking copious free time schedule… We’ll see.

    -Brandon

    P.S. Isn’t it annoying when you’re typing in a web form and try to hit escape to enter command mode, only to remember Netscape doesn’t support modal editing. =(

  2. Johan Ihren says:
    September 13, 2001 at 11:16 am

    Since you wrte about Big Brother previsously it seems natural
    to make some sort of comparision of relative strengths and
    weaknesses.

    • RevDigger says:
      September 14, 2001 at 10:17 am

      I’ll field this one.

      I’ve been running BB for a couple years now and it’s been pretty decent. The overall impression that one gets from BB is that it is a giant hack. It’s all shell scripts and convoluted pipes and greps and stuff. I’m sure the author was an over worked sysadmin who needed to take a day off, so he cobbled together a bunch of hacks, and it worked, so he let his friends use it too.

      Netsaint, however, gives one the impression that it was actually designed. Things are well organized, funtions are in plugins, etc. I have found Netsaint allows me a lot finer control over it behaviour, but then it also required a lot more work to get going properly. Netsaint seems to do fewer weird things.

      By the way, I have the original PSDs of the BSD-Style graphics, if anyone want to play with them…

  3. Chris says:
    September 15, 2001 at 6:40 am

    Has anyone gotton the netsaint.cmd file for external commands working [FreeBSD 4.3_REL]?

    I follow the directions to the T, but when Apache [as nobody] creates the netsaint.cmd, the sticky bit for the group gets ignored, thus the file’s group can’t do squat and NetSaint never processes it.

    I sifted through the newgroups, but found little if nothing on it..

    Thanks

  4. Dan Langille says:
    September 23, 2001 at 5:22 am

    The FreeBSD NetSaint port now contains the graphics.

  5. RevDigger says:
    September 27, 2001 at 6:51 am

    Netsaint comes with lots of tools that monitor remote services, like the HTTP and SMTP checkers. But how do you check remote status properties like free disk, or load average? Well of course, people have thought of this.

    The "Addons: section of the Netsaint download page:

    <A HREF="http://www.netsaint.org/download/">http://www.netsaint.org/download/</A&gt;

    has at least 3 different packages for doing this. I looked at:

    netsaint_statd v2.11
    NetSaint Remote Plugin Executor (NRPE) 1.2.4
    NetSaint Remote Plugin Executor/Perl (NRPEP) 0.2

    For various reasons, I went with netsaint_statd, but if you have had a good experience with any of the others, let me know. netsaint_statd consists of an RPC daemon that runs on the remote machine, and a series of plugins with which the Netsaint server can query it. For security, it implements an allowed-hosts list and will only report a limited set of data.

    After untaring and reading through the brief INSTALL doc, I simply put a copy of the rpc daemon on each of the servers from which I wanted reports. I decided that /usr/local/libexec/ was a decent place for it. The RPC daemon is a Perl script, and you will want to edit it to add your Netsaint server to the list of addresses allowed to query it. Since I seem to be restarting these servers all the time (every 3 – 6 months), I dropped a little init script in /usr/local/etc/rc.d/. It looks like:

    #!/bin/sh
    /usr/local/libexec/netsaint_statd

    Once I had those all running I returned to my Netsaint server and dropped the new plugin in my Netsaint plugins directory. Actually I put them in a sub directory,

    /usr/local/libexec/netsaint/netsaint_statd/

    to keep things tidy. Last thing to do was to let Nesaint know about its new toys.

    First I added the following lines to Netsaint’s commands.cfg:

    # netsaint_statd remote commands
    command[check_rload]=$USER1$/netsaint_statd/check_load.pl $HOSTADDRESS$ $ARG1$ $ARG2$ $ARG3$
    command[check_rprocs]=$USER1$/netsaint_statd/check_procs.pl $HOSTADDRESS$ $ARG1$ $ARG2$ $ARG3$ $ARG4$
    command[check_rusers]=$USER1$/netsaint_statd/check_users.pl $HOSTADDRESS$ $ARG1$ $ARG2$ $ARG3$
    command[check_rdisks]=$USER1$/netsaint_statd/check_all_disks.pl $HOSTADDRESS$ $ARG1$ $ARG2$ $ARG3$ $ARG4$
    command[check_rnamed_proc=$USER1$/netsaint_statd/check_named_proc.pl $HOSTADDRESS$ $ARG1$ $ARG2$ $ARG3$ $ARG4$

    which I mostly just copied from the netsaint_statd README. Lastly, I added services I wanted to monitor to my hosts.cfg and restarted my Netsaint process. That’s it, really.

    Another nice thing is that adding your own plugins looks like it would be easy to hack in.

  6. Dan Langille says:
    October 4, 2001 at 2:49 am

    Here’s what I had to do for checking https:

    hosts.cfg:

    service[dev]=HTTPS;0;24×7;3;2;1;freebsd-admins;120;24×7;1;1;1;;check_https!

    commands.cfg:

    command[check_https]=$USER1$/check_http –ssl $HOSTADDRESS$

    I recommend testing from the command line to ensure your check_http can do this:

    $ /usr/local/libexec/netsaint/check_http –ssl 10.0.0.1
    HTTP ok: HTTP/1.1 200 OK – 1 second response time

    Make sure you have https not running and http running. Then test it again:

    $ /usr/local/libexec/netsaint/check_http –ssl 10.0.0.1
    Connection refused by host

    $ /usr/local/libexec/netsaint/check_http 216.187.106.227
    HTTP ok: HTTP/1.1 200 OK – 1 second response time

    Then you know it’s working and it’s actually checking https and not http.

    You might, and I stree might have to recompile the plugins. My memory is hazy but I don’t think I had to do this in order to get the –ssl option to work:

    cd /usr/local/etc/netsaint/plugins/netsaint-plugins-1.2.9-4
    make clean
    ./configure
    make
    cp check_http /usr/local/libexec/netsaint

    I’ll verify this and update as necessary.

    • RevDigger says:
      October 13, 2001 at 6:17 am

      I forget if this is standard or something I did myself, but my commands.cfg has this:

      command[check_ssl]=$USER1$/check_http -H $ARG1$ -S -C $ARG2$

      -H is for the host name, in case it’s a VHost and that is pertinent. -S tells it to use ssl. And -C is the number of days before the cert expires to start warning me. That’s right, it checks the expire date on the cert for you. Too cool.

      So in my hosts.cfg I have something like:

      service[lisa]=SSL;0;24×7;3;2;1;unix-admins;180;24×7;1;1;1;;check_ssl!secure.myhost.com!14

      Works great. In my Netsaint Status Detail, I get, "Certificate will expire on 07/03/2002 23:59."

  7. Raul Viikna says:
    October 28, 2001 at 8:35 pm

    (posted by Dan Langille)

    About netsaint. If it complains "printf: illegal format character" after
    startup and e-mail notifications have no body then look at:

    <A HREF="http://www.geocrawler.com/archives/3/1155/2001/5/0/5811870/">http://www.geocrawler.com/archives/3/1155/2001/5/0/5811870/</A&gt;

    <A HREF="http://www.geocrawler.com/archives/3/1108/2001/6/50/6064473/">http://www.geocrawler.com/archives/3/1108/2001/6/50/6064473/</A&gt;

  8. Dale Offret Jr. says:
    March 29, 2002 at 12:02 am

    I have followed your article, by the way it is great, and setup my cfg files and such. I do have netsaint running but thttpd reports and internal error.

    500 Internal Error

    There was an unusual problem serving the requested URL ‘/cgi-bin/netsaint/statusmap.cgi?host=all’.

    thttpd/2.21b 23apr2001

    Anyone who has this working please let me know.

    Thanks,

    Dale Offret Jr.

    • Dan Langille says:
      March 29, 2002 at 5:28 am

      The only people who might see your message here are those that are reading the article. Why don’t you click on FORUMS and post your problem on the Support forum where many more people will see it.

  9. wtpson says:
    October 20, 2002 at 2:27 am

    Hello,

    I read the netsaint documentation and put as many things as possible into MySQL database. To do this, you have to make these steps
    cd /usr/ports/net/netsaint/

    make CONFIGURE_ARGS+=’–with-mysql-xdata –with-mysql-lib=${LOCALBASE}/lib/mysql’ all install

    mysqladmin create netsaint
    cat /usr/ports/net/netsaint/work/netsaint-0.0.7/contrib/database/ | mysql netsaint

    mysql mysql
    —- cut here —–
    insert into user (host,user,password) values (‘localhost’,’netsaint’,password(‘your_password_here’));
    insert into db values (‘%’,’netsaint’,’netsaint’,’Y’,’Y’,’Y’,’Y’,’Y’,’Y’,’Y’,’Y’,’Y’,’Y’);
    flush privileges;
    \q
    —- cut here —–

    cd /usr/local/etc/netsaint/
    edit following files:
    resource.cfg to activate DB * DATA
    nscgi.cfg to activate DB * DATA
    nscgi.cfg to *CHANGE* command for checking netsaint alive (read below)
    netsaint.cfg to comment (#) comment_file

    Why you have to cahnge netsaint_check_command? Well, netsaint uses:
    /usr/local/libexec/netsaint/check_netsaint /usr/local/var/netsaint/status.log 5 ‘/usr/local/bin/netsaint’

    with default. When you activate all possible data with DB, there is no /usr/local/var/netsaint/status.log file, so you will receive error message, that possible netsaint is not running. And then, some command could not be run. So to make netsaint daemon satisfied, I change netsaint_check_command with value:

    /usr/local/libexec/netsaint/check_procs -w 2 -c 2 -C ‘/usr/local/bin/netsaint’

    And everything is OK.

    What are advantages of using DB? Well, there are some:

    * it is very, very easy to modify 2D or 3D status map (just ine update in hostexinfo table)
    * you save disk sapce for some logs (status.log)
    * it’s much quicker to get some info from DB.
    * after update in hostexinfo table there is no need to reload netsaint with new configuration.

    Unfortunatelly in current release (0.0.7_2,1) there no possibilities to move hosts.cfg to database. but maybe in the future… Who knows…

    I hope, that I missed nothing, so you could repeat this steps without any error.

    Please feel fre to post me, if you have any problems

    Greeting
    Krzysztof Stryjek

    • wtpson says:
      May 16, 2003 at 10:02 am

      Hello,

      Some time ago I’ve posted a PR to include Database support in netsaint and nagios ports. This PR was closed (support included) about 2-3 month ago. So now it is much easier to do it.

      Best regards
      Krzysztof Stryjek