Sep 262012

Its good to have more information and installing, configuring or managing systems. As DevOps I find it often necessary to make things easier, better, faster even on system level not just in application coding.

I highly recommend adding RPMForge to your CentOS. Instructions are here. Centos RPMForge

  • multitail – the better tail

I find myself often staring at log lines. But I hate that fact that on terminals they all look the same and its hard to tell whats changed. Specially if you are looking at logs that scroll at 20 lines per second.
I found multitail by accident when googling for pausing my “tail -f”. This is before the days of StackOverflow and Serverfault. Those days I was compiling multitail for every system as there was no packagaes available. Its fairly simple so I did not bother looking elsewhere. Good news. if you added RPM Forge it is  just

#yum install multitail

test it with any log

#multitail -c /var/log/httpd/access.log

press P to pause. Press Q to quit.

RTFM to see how you can actually add windows in your terminal to watch more than one log file side by side, you can press F1 when the program is running to get to quick docs. For example you may want to watch the HTTP Access and Error logs together.

  • htop – flavorful top

I do not know or care what is the “h” for in htop. It is definitely better than top. Here’s a fricking screenshot.

You can use the menu to configure quite a few things in the view. Unfortunately it does not show any network related info as far as I remember.

It matters not as network has quite a few tools not only in RPMForge but in base as well. ntop is an ok tool but not available in RPMForge or Base. There is vnstat that collects stats on network interface but I would avoid these tools on production as they tend to use CPU and Disk space which can be bad for high traffic servers. Unless ofcourse you have a very good reason to use these. I have a favorite one which I will get to later on.

Some networking related tools useful for admins and devops. Most are based off of libpcap so its best to install it from base first.

  •  dnstop – a top for DNS queries

It was once a few years ago when I was much too green in production environment and fresh out of uni that I realized DNS attacks can actually give you not just a bad day but a whole week. They can be so hard to figure out.
On positive note if you are running DNS queries in your web applications  then this tool is useful as well. Ref: 

It does not much except give you statistics on DNS related stuff on your system on any interface. basic usage
#dnstop eth0

Where eth0 is your interface, whatever iface you use.  This utility is useful to detect DNS query based attacks. Also useful for debugging programs that may be leaking bad domain name lookups etc. See the docs for details on this. Its is quite normal to see a communication with your primary DNS as per the nameserver in /etc/resolve.conf. But there should be no queries where source is an external IP unles syou are running a DNS server. And if you are running one then this utility is just indispensable.

*  darkstat – sheds light on network and is lightweight
although I wonder why the author Emil Mikulic called it darkstat.  This is better than ntop, so dont even waste time installing ntop and blah.

I would write a guide fo rit but a neat quickstart guide is available here.

In any case here is my quickstart guide

#yum install darkstat

#darkstat -i eth0 -p 8080

now browse to port 8080 or whatever port you set and see your network stats.  The system is not only easy to use, it is useful and have been able to spot network attacks quickly and easily to block those nasty IPs and neighbors. I also find it useful for debugging issues with network apps.



 Posted by at 3:22 am