Knowledge Is Power
|
Knowledge Is Power |
|
UNIX and Linux systems run the syslog daemon by default which means they can act as a "loghost" which allows them to accept log messages from other devices over the network. The UNIX or Linux system will then write these messages to log files stored on a local hard-drive. Most Cisco switches and routers can send their log messages to a UNIX or Linux server that is operating as a loghost. This page shows you how to use a Perl shell script featured below to monitor the Cisco log files and send an e-mail alert in the event an error occurs on one of your Cisco devices. Taken as a whole, this setup can offer SNMP-like monitoring of Cisco hardware without the hassle of setting up a full-blown SNMP system. Other UNIX systems can be set up to use a loghost nd if you do some checking, you may find that other devices and OSs running on your network can as well.
Although this procedure on this page was written for a Solaris x86 box, any UNIX or Linux system should be able to act as a loghost. Once you have a UNIX or Linux system set up (including networking and Sendmail), you'll need to do the following:
All of these steps are detailed below.
- configure your UNIX/Linux system to act as a loghost
- configure your Cisco switches and routers to use a loghost
- set up the shell script to check the log files (example below)
- set up a cron job to regularly run the script
If you didn't use the Trying Sun Solaris for Intel x86 companion page to set up your UNIX/Linux system, you'll need to check your hosts file to make sure it contains the loghost keyword. The hosts file created on our Solaris system looks like this:Next you want to open the /etc/syslog.conf file in the text editor and add the following line near the bottom of it, pressing Enter at the end of the line, and save the file:Note that you have to press the Tab key five times after entering "debug" before you enter the path to the log file. The "local7" is the "server facility" most Cisco devices use by default.
local7.debug /var/log/cisco.log
The final step in setting up the server is creating the log file that was specified above, as well as a history file that the monitoring shell script will use. Type in the following commands to create the empty log files and make them writable:
After you've entered these commands reboot your servertouch /var/log/cisco.log
touch /var/log/cisco.history
chmod 666 /var/log/cisco.*( init 6 ).
That takes care of the server. It's ready to receive log entries from your Cisco devices so now we have to set those up.
There are two different flavors of the IOS used in Cisco switches. Unfortunately there's no rhyme or reason as to which switch model uses which IOS. For instance, our 2900 switches use the traditional Cisco IOS, while our 2948G switches use the "Set/Clear" IOS. These two IOS flavors use different commands at the CLI (Command Line Interface).
As of this writing, the traditional Cisco IOS is at rev 12.x, while the Set/Clear IOS is at rev 5.x. If you type inshow version at the CLI prompt the version number will be displayed and you will be able to tell which IOS you are dealing with.
You may also be able to use the CWI to set up logging on your switches. CWI is the Web interface available on many Cisco models. To see if it is enabled, simply fire up a browser and on the "Address" line enter the IP address of your switch. On our switches the logging options are under the Fault link.
Note: The above brings up an important point. Out of the box, Cisco switches don't have an IP address. You have to assign one to them (along with a subnet mask and default gateway). If you have not already done so, use the software setup manual that came with your switches to assign IP addresses to them. The IP address is used to identify the switch in the log file entries.The following procedures assume you already know how to use the Console port and Hyperterminal, or how to Telnet into your routers or switches, to get at the privileged-exec (#) CLI prompt.
Traditional Cisco IOS
I think all Cisco routers use the traditional Cisco IOS but I'm no expert on Cisco hardware. If you have any switches that also use the traditional IOS, these are the commands that you need to enter at the privileged-exec prompt (#):
- config term
(Your prompt will change to include the word "config")- logging <ip address of your Solaris box>
- logging trap errors
- service timestamps log datetime
- logging on
Press Ctrl-Z to exit out of config mode. Use the No form of the above command to turn logging off(i.e. no logging on )
Set/Clear IOS
If you have any switches that use the Set/Clear (Cat) IOS, these are the commands that you need to enter at the privileged-exec prompt (#):
- set logging server <ip address of your Solaris box>
- set logging server severity 3
- set logging timestamp enable
- set logging server enable
Thats it! As of now your switch/router will start logging messages to your loghost system. Repeat the above for each of the switches or routers you want to monitor.
You can enter theshow logging command at the privileged exec prompt of either IOS to verify that logging is set up correctly. The display you see will depend on which flavor of IOS you have (and probably which version within each flavor). Look near the top of the listing that theshow logging produces for lines starting with following for your flavor of IOS, where x.x.x.x is the IP address of your Solaris box:
Traditional Cisco IOS
trap logging: level errors
logging to x.x.x.x
Set/Clear IOS
Logging server:
(x.x.x.x)enabled Server facility:
Server severity:
LOCAL7
errors(3)
Message Types
There are six different levels of message severity that your Cisco switches/routers will produce. The lower the number the worse the condition of your hardware. They are, listed from highest to lowest severity (lowest to highest number):
With the above IOS commands, only messages of severity level 3 or worse will be written to the loghost. You can alter this if you wish, but the shell script below will e-mail an alert to you if any message is written to the loghost.0 - Emergency
1 - Alerts
2 - Critical
3 - Errors
4 - Warnings
5 - Notification
6 - Information
7 - Debug
This is an important point to remember since you may think that your logging setup isn't working because nothing ever appears in your log file. In actuality, you probably just have healthy equipment. We'll show you how to test your logging setup below so that you'll know for sure.
You may have to adjust the severity value in the IOS commands depending on the flavor of IOS your devices run and what role your devices play in your network. With the Set/Clear IOS, you could probably use a value of 4. With the traditional IOS however, even the value of 3 (errors) may generate numerous alerts as even link state changes (a system being turned off) can generate a level 3 log event. Losing a link is a problem with a router, but if you have workstations connected to switches running the traditional IOS (such as our 2900s), you may want to even use level 2 (critical) if you find you are receiving alters due to link state changes. Naturally, you can use different values on each individual device depending upon its' role.
Logging errors doesn't do us much good if no one ever looks at them. And having to look at another log file is the last thing most network administrators need. So we'll automate the process using a shell script to keep an eye on the log file and let us know if anything shows up.
I used Perl to write this shell script due to it's inherent ability to deal with text, and because with Perl you don't have to worry about differences in commands between the different shells. Perl is a very easy scripting language to learn and in addition to it being very useful for shell scripts, it is also the language of choice for most CGI scripts used on Web sites. There is an introduction to it on the How To Use Your CGI-BIN companion page.
The script will do the following:
The script is called logchek and the text of it follows, but you can also click here to open it in a separate window so you can do a "Save As". But don't keep the .txt when you save the file. Save it simply as logchek to the /usr/local directory.
- Check the cisco.log file for any entries (by checking the file's size)
- Exit the script if no entries exist (file size = 0 bytes)
- If there are entries:
- Copy the entries to the cisco.history file
- Clear the cisco.log file
- Send an e-mail alert to the designated recipient telling them to check the cisco.history file
- Exit the script
You'll need to do the following with this script:#!/usr/bin/perl ########################################################################## # logchek - Sample PERL shell script for monitoring/reporting new errors # # to a remote log file for Cisco devices. chmod to 755 # # Written by Keith Parkansky - 7/2001 # # Made available STRICTLY "AS IS". # # This script is intended to be used on a UNIX loghost as # # outlined at https://www.parkansky.com/bdl/bdlogcis.htm # # CONDITIONS OF USE: # # 1. Acceptance that no warranties are expressed or implied. # # 2. USE IS AT YOUR OWN RISK. # # 3. Under no circumstances will Keith Parkansky or Cox # # be held liable for any consequential, incidental, or # # any other damages whatsoever resulting from the use of # # this code or derivatives of it. # # 4. Any redistribution of this script is only allowed if # # the script remains unaltered in any way including all # # messages, comments, statements, and code segments. # # Use of this file in any way constitutes acceptance of the # # above conditions. # # # # This script is intended to be run regularly as a cron job. # # It operates as follows: # # - Initialize variables # # - Check log file for entries (by checking file size) # # - Exit script if no entries exist # # - If entries exist: # # * Read in log entries and clear log file # # * Write entries to history file # # * Send an e-mail alert to a specified recipient(s) # # * Exit script # ########################################################################## # ENTER VALUES FOR THE FOLLOWING VARIABLES ! #-------------------------------------------- $hostname = 'myhost'; # Your loghost's hostname $mailprog = '/usr/lib/sendmail -t'; # Path to the sendmail program $sendto = 'me@mydomain.com'; # E-mail address of alert recipient #-------------------------------------------- # These file names should match what's in your /var/log directory # Use touch to create them if necessary and chmod to 666 $log = '/var/log/cisco.log'; $history = '/var/log/cisco.history'; $logsize = (-s "$log"); # Check for entries in log file if ($logsize > 5) # If entries exist { # Open log file and read lines into a single dim array open (LOG,"<$log") || die 'Could not OPEN log file'; @loglines = <LOG>; close (LOG); # Clear out the log file open (LOG,">$log") || die 'Could not OPEN log file'; print LOG "\n"; close (LOG); # Write log entries into the history file open (HISTORY,">>$history") || die 'Could not OPEN history file'; foreach $newhistoryline (@loglines) { print HISTORY $newhistoryline; } close (HISTORY); # E-mail an alert to the sysadmin open (MAIL, "|$mailprog") || die 'Could not open mail program'; print MAIL "To: $sendto\n"; print MAIL "From: $hostname\n"; print MAIL "Subject: CISCO LOG ALERT\n\n"; print MAIL "Check /var/log/cisco.history file for new errors."; close (MAIL) } exit;
You can enter multiple e-mail addresses in between the single quote marks simply by separating them with commas if you want to send alerts to several people.
- Save the file to the /usr/local directory
- chmod it to 755 using the command:
chmod 755 /usr/local/logchek - Enter the appropriate values for:
- the path to your Perl installation (the very first line)
(the value in the script code is correct for Solaris x86 systems)- your loghost system's host name
- the path to your Sendmail (or other SMTP) program
(the value in the script code is correct for Solaris x86 systems)- the e-mail address of the recipient(s) of alert messages
If your pager or cell-phone is e-mail enabled you could also enter the e-mail addresses for those devices as well. But if you do, you may not want to set the script to run as a cron job in the wee hours of the morning.
Note that the cisco.log file is used as sort of a "trigger" file that only stores log messages temporarily. The cisco.history file is the file that will contain the error messages that you want to look at should you receive an alert message.
At this point you'll want to test your system to verify that the Cisco devices can log their messages to your loghost and that the shell script works.
Testing Logging
You only need to try this test on one of your Cisco devices since demonstrating that one device can log to the loghost verifies that any similarly-configured devices will be able to also. In order to get the device to try and log messages to the loghost without having to wait for a long period of time, you'll need to temporarily change the logging severity level of the device. The busier your network is the sooner a loggable message will be generated.
Traditional Cisco IOS
If the device your testing uses the traditional IOS, enter these commands at the privileged-exec prompt:
- config term
- logging trap informational
(Re-issue the logging trap errors command after testing)
Set/Clear IOS
If the device your testing uses the Set/Clear IOS, enter this command at the privileged-exec prompt:
- set logging server severity 6
(Re-issue the set logging server severity 3 command after testing)
Back on your loghost server, go to a shell prompt and type in the following commands:and note the file size of the cisco.log file. Keep re-entering the command until you see the file size increase. When it does, issue the following command to view the messages logged:cd /var/log
ls -laFOnce you have messages in the cisco.log file, you can return to the Cisco device and reset the logging severity level back to 3.cat cisco.log
If you get tired of waiting for messages to appear, locate a computer that is connected to one of the ethernet ports on the switch and turn its' (the computer's) power off (not just a re-boot) and verify that the link indicator for the port on the switch that the computer is connected to goes out. After a few minutes, turn the computer back on and verify that the link indicator for the port comes back on. This should change the link state of the switch port and generate a loggable message.
If you don't get any messages appearing in your log file on the loghost it's time to do some troubleshooting. Possible causes are:
- the "facility" setting in your Cisco device is not set to "local7" (this is the factory default)
- the entry in the syslog.conf is not correct (are you sure you spelled things correctly and pressed the Tab key 5 times when entering the line?)
- the cisco.log file is not writable (did you remember to chmod the file to 666?)
Testing The Script
Once you've got messages in the cisco.log file it's time to test out the shell script. Assuming you're still at the shell prompt from above, type in the following command:If you don't see anything that's a good sign. You shouldn't since there is no console output in this script. Check to see if the messages that were in the cisco.log file got transferred over to the cisco.history file. At the shell prompt type in:./logchekand check to make sure that the log messages you saw earlier in the cisco.log file are now displayed on the screen (indicating that they are now in the cisco.history file). You may also want to type in:cat cisco.historyand note the file sizes of the two cisco log files. The cisco.log file size should only be 1 byte while the cisco.history file should have grown in size. You should also receive an e-mail with "CISCO LOG ALERT" in the subject line.ls -laF
If the messages didn't get transferred you've got some troubleshooting to do. Make sure of the following:As mentioned earlier, the procedures detailed here are for a Solaris x86 standard installation. Adjustments to paths, etc. may need to be made for other flavors of UNIX and Linux.
- the logchek script file has the 755 permissions
- the path to your Perl installation is correct (the very first line of the script)
- the path to your mail program is correct
You're almost done. The final step in setting up your Cisco device monitoring is to set up the logchek script to run as a cron job to routinely check the log file and alert you if necessary. Again, the following instructions are for a Solaris x86 system but this cron job can be set up on any UNIX system.
Cron is the daemon that runs in the background waiting to execute jobs and it is set to start at system boot-up by default during the Solaris installation. You don't have to do anything to get cron running. The jobs it executes are listed in a crontab file which is edited using the crontab utility.
The following command will list the contents of your current crontab file:
crontab -l
You'll want to add an entry to the crontab file for the logchek shell script. The format of the file is basically:
<when-to-run values> <what to run>
The "When"
The "when to run" is a field which consists of five space-delimited values in the following order:
You can use the asterisk (*) to specify all values for any given entry. For example, to run a job every Saturday at 11:15 pm you would use the following values:
- Minutes past the hour (0 to 59)
- Hour of the day (0 to 23)
- Day of the month (1 to 31)
- Month of the year (1 to 12)
- Day of the week (0=Sunday to 6=Saturday)
15 23 * * 6
Be careful with these values. You'd rarely want to enter a number for the "Day of the Month" and the "Day of the Week". For example, if you entered:
15 23 3 * 6
cron would only run the job when the 3rd falls on a Saturday.
How often you want to check for log errors from your Cisco devices is entirely up to you. Daily, hourly, every four hours, whatever. In addition to the frequency of the checking, you'll also have to decide on the times it should be checked. As mentioned earlier, if you're going to have the alert messages e-mailed to your pager you probably don't want to schedule a cron job to run at 3 am.
You can enter multiple values for each entry by separating them with commas. We set up cron jobs to check the logs twice a day, every weekday, at noon and again at 5 pm. This required the following values:
0 12,17 * * 1,2,3,4,5
Remember that the space is the delimiter between the five entries.
The "What"
What you want to execute is basically anything you can enter at a shell prompt. Any command, series of commands including pipes and redirects, shell script name, etc. Since we want to run the logchek shell script, which we saved to the /usr/local directory, our crontab entry ends up looking like this:
0 12,17 * * 1,2,3,4,5 /usr/local/logchek
Note that only a space separates the command that is to be executed from the last of the "when to run" values.
So now that we know what our entry will be, we have to use crontab to enter it into the crontab file. At the shell prompt, enter:
crontab -e
This will fire up your default text editor with the current crontab file automatically loaded. Simply enter your new crontab entry and close the editor. Answer Yes when prompted to save the file. You can check to make sure your entry was added to the crontab file by entering the following command at the shell prompt:
crontab -l
That's it! You're all done! Happy monitoring!
Attention Network Administrators
Monitor the temperature, humidity, and AC line power in your server room 24/7 with a Sensaphone. Cost of the Sensaphone 1104 unit and optional sensors is only around $400! It will dial pagers and deliver a synthesized voice message to cell phones.
I do not have any financial interest is mentioning this. I simply want to pass along a good thing when I find it (and I was searching for something like this for weeks).
Another gem is IP Sentry. Monitor your NT and UNIX/Linux servers (file, application, and Internet servers) 24/7 with e-mail and pager alerts. Monitors individual services on NT servers and individual protocols (http, ftp, etc) on Internet servers. Only $99 and you can download a free trial version! I likewise have no financial interest in this product.
If you're administrating NT/2000 networks you may want to check out Hyena. It basically takes User Manager, Server Manager, and a host of other administration utilities and rolls them all into on GUI interface. The Enterprise edition allows you to manage Exchange accounts (even manage Exchange accounts automatically when managing user accounts), all for $270. ($199 without the Exchange piece). You can download a free trial of this also. And you guessed it, no financial interest here either.
IN NO EVENT WILL KEITH PARKANSKY BE LIABLE TO ANY PARTY (i) FOR ANY DIRECT, INDIRECT, SPECIAL, PUNITIVE OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF PROGRAMS OR INFORMATION, AND THE LIKE), OR ANY OTHER DAMAGES ARISING IN ANY WAY OUT OF THE AVAILABILITY, USE, RELIANCE ON, OR INABILITY TO USE THE INFORMATION, METHODS, HTML OR COMPUTER CODE, OR "KNOWLEDGE" PROVIDED ON OR THROUGH THIS WEBSITE OR ANY OF ITS' ASSOCIATED DOCUMENTS, DIAGRAMS, IMAGES, REPRODUCTIONS, COMPUTER EXECUTED CODE, OR ELECTRONICALLY STORED OR TRANSMITTED FILES OR GENERATED COMMUNICATIONS OR DATA EVEN IF KEITH PARKANSKY SHALL HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND REGARDLESS OF THE FORM OF ACTION, WHETHER IN CONTRACT, TORT, OR OTHERWISE; OR (ii) FOR ANY CLAIM ATTRIBUTABLE TO ERRORS, OMISSIONS, OR OTHER INACCURACIES IN, OR DESTRUCTIVE PROPERTIES OF ANY INFORMATION, METHODS, HTML OR COMPUTER CODE, OR "KNOWLEDGE" PROVIDED ON OR THROUGH THIS WEBSITE OR ANY OF ITS' ASSOCIATED DOCUMENTS, DIAGRAMS, IMAGES, REPRODUCTIONS, COMPUTER EXECUTED CODE, OR ELECTRONICALLY STORED, TRANSMITTED, OR GENERATED FILES, COMMUNICATIONS, OR DATA. USE OF THIS SITE CONSTITUTES ACCEPTANCE OF ALL STATED TERMS AND CONDITIONS.