Netdump Server Setup

This wiki describes the steps to set up a netdump server minimizing any risks and downtime.

Netdump is the dump utility used by default in Centos 4.5.
It has some restrictions, basically not all network cards are supported.
I have successfully used it with a Broadcom TG3

Netdump setup

**Setup Netdump Server:

1. Check to see if the netdump-server is installed:

rpm –qa | grep –i netdump-server

If it is not already installed, install the netdump-server package.

yum install netdump-server

Netdump creates a ssh user which is used to transfer the dump and which authenticates using a passwordless rsa certificate.
Although this is not really a security risk, I like at least changing the name of the user netdump runs as. Even with more reason if this server can be accessed by untrusted clients.

2. Delete the user netdump and create a user cr4shdump

userdel netdump
adduser cr4shdump -d /var/crash
chown -R cr4shdump:cr4shdump /var/crash

3. Set a very difficult password for the cr4shdump user.

passwd cr4shdump

Check the name of the authorized file you are using in your system

grep AuthorizedKeysFile /etc/ssh/ssd_config

4. Change the key for the cr4shdump user

su - cr4shdump
cd .ssh
cp authorized_keys2 authorized_keys2.`date +%d%b%g`
cp authorized_keys authorized_keys.`date +%d%b%g`
cp authorized_keys2 authorized_keys

5. Substitute the content of authorized_keys with the content of the file /etc/sysconfig/netdump_id_dsa.pub

Make sure after modifying that the file the permissions and owner are like this

-rw-— 1 cr4shdump cr4shdump 616 Mar 6 08:50 authorized_keys

6. Add the cr4shdump user to sshd AllowUsers directive (In case you need to)

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.`date +%d%b%g`
vi /etc/ssh/sshd_config
/etc/init.d/sshd restart

7. Modify the init script

Change:
daemon —user netdump /usr/sbin/netdump-server —daemon $SERVER_OPTS

To:
daemon —user cr4shdump /usr/sbin/netdump-server —daemon $SERVER_OPTS

8. Configure and start the service.

chkconfig netdump-server on
service netdump-server start

Now we are going to modify the firewall, first of all take preventive measures in case something goes wrong

9. Take the time on the server

date

**10. Edit crontab and put this line for 10 minutes later than the actual time, this will allow us to revert the firewall in case we lock ourselves out **

crontab -e
25 * * * * /etc/init.d/iptables restart

11. Edit the new iptables file

cd /etc/sysconfig
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.‘date +%d%b%g`
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.`date ’+%d%b%g'`.new
vi /etc/sysconfig/iptables.`date +%d%b%g`.new

-A INPUT -m udp -p udp -s 192.168.75.229 —dport 6666 -j ACCEPT
-A INPUT -m udp -p udp -s 192.168.75.233 —dport 6666 -j ACCEPT

12. Upload the new firewall

iptables-restore < /etc/sysconfig/iptables.`date +%d%b%g`.new

13. Test if the port is open

nc -v -u -z -w 3 server 6666

14. Check that you can open a new ssh session to the box from your computer

ssh mario@server

15. Check that any other used ports are working from your computer.

nc -v -z -w 3 server 25
nc -v -z -w 3 server 993
nc -v -z -w 3 server 995

16. Make iptables changes permanent.

mv /etc/sysconfig/iptables.`date +%d%b%g`.new /etc/sysconfig/iptables

17. Edit crontab and commment out the restart line

crontab -e
#25 * * * * /etc/init.d/iptables restart

Setup Netdump client (the machine that is crashing)

1. Check to see if the netdump-0.6.11-3 is installed.

rpm –qa | grep netdump

If it's not already installed, install the netdump client package

yum install netdump

2. Find out the mac address of the server and if the client can resolv it.

arp -a

2. Edit the file /etc/sysconfig/netdump.

cp /etc/sysconfig/netdump /etc/sysconf/netdump.`date +%d%b%g`
Add a line NETDUMPADDR=x.x.x.x in which you specify the address of the netdump server.
NETDUMPADDR=192.168.75.230 (address of the server)
NETDUMPMACADDR=00:A0:81:47:C1:FA

3.Change the init script

cp /etc/init.d/netdump /etc/init.d/netdump.`date +%d%b%g`
vi /etc/init.d/netdump
Change:
ssh -x -i /etc/sysconfig/netdump_id_dsa netdump@$NETDUMPADDR […]

To:
ssh -x -i /etc/sysconfig/netdump_id_dsa cr4shdump@$NETDUMPADDR […]

Change:
ssh -x netdump@$NETDUMPADDR cat '»' /var/crash/.ssh/authorized_keys2

To:
ssh -x cr4shdump@$NETDUMPADDR cat '»' /var/crash/.ssh/authorized_keys2

4. Configure and start the service

chkconfig netdump on
service netdump start

5. Enable kernel panic for this session and future ones:

echo 1 > /proc/sys/kernel/sysrq
cat /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/panic_on_oops
cat /proc/sys/kernel/panic_on_oops
cp /etc/sysctl.conf /etc/sysctl.conf.`date +%d%b%g`
vi /etc/sysctl.conf
kernel.sysrq= 1
kernel.panic_on_oops= 1

6. Check var/log/messages

tail -n20 /var/log/messages

7. Check to see if netlog is working

echo h > /proc/sysrq-trigger
tail /var/log/messages

8. Open port 6666 udp connections outbound in the firewall /etc/sysconfig/iptables

cd /etc/sysconfig
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.‘date +%d%b%g`
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.`date ’+%d%b%g'`.new
vi /etc/sysconfig/iptables.`date +%d%b%g`.new
-A OUTPUT -d 192.168.75.234 -p udp -m udp —dport 6666 -j ACCEPT

9. Take the time on the server

date

10. Edit crontab and put this line for 10 minutes later than the actual time.

crontab -e
25 * * * * /etc/init.d/iptables restart

11. Upload the new firewall

iptables-restore < /etc/sysconfig/iptables.`date +%d%b%g`.new

12. Check that you can open a new ssh session to this box from your computer

ssh mario@crashingServer

13. Make iptables changes permanent.

mv /etc/sysconfig/iptables.`date +%d%b%g`.new /etc/sysconfig/iptables

14. Edit crontab and commment out the restart line

crontab -e
#25 * * * * /etc/init.d/iptables restart

To test the setup

1. In the client run

nc -v -u -z -w 3 server 6666
echo h > /proc/sysrq-trigger
tail /var/log/messages

2. In the netdump server check that a directory with the ip address of the client has been created in /var/crash

To test the setup (Crashing the client)

1. On the client check to see if netlog is working

echo h > /proc/sysrq-trigger
tail /var/log/messages

2. In the netdump server check that a directory with the ip address of the client has been created in /var/crash

3. Sync all mounted file systems

echo s > /proc/sysrq-trigger

4. Crash the system

echo c > /proc/sysrq-trigger

On the netdump server's directory (/var/crash) two files should be created: log and vmcore.

Optional, Set up /proc/sys/kernel/panic What ever number you put here will be the time in seconds, the kernel will wait until restarting the machine automatically. /proc/sys/kernel/panic doesnt work with netdump at the same time, you either enable one or the other.

For extra security, when you have everything working, replace the certificate that comes with netdump with your own one

In the server run:

su - cr4shdump
cd .ssh
cp authorized_keys2 authorized_keys2.`date +%d%b%g%s`
cp authorized_keys authorized_keys.`date +%d%b%g%s`
ssh-keygen -t rsa (Accept the default values)
cat id_rsa.pub > authorized_keys

Copy the file id_rsa from the server to the crashingServer in /etc/sysconfig/netdump_id_dsa

back to Linux documents