Re: [nv-l] Problems with SNMP monitoring

[Gareth Holl <gholl@us.ibm.com>]

Bill,

netmon sets one of the interfaces on
the core router to be the SNMP address to be used as the status polling
address as you probably know. And it uses this address to obtain the status
of all interfaces on the core router. Is this SNMP address changing regularly
? Could it be changing to an interface that might be isolated from NetView
? Can you observe the SNMP address to see if it is changing at all from
poll to poll, or at least every 32 hours. This is all I can think of initially
but I'll keep thinking about the problem you've described.

Cheers,

Gareth Holl
Staff Software Engineer
gholl@us.ibm.com

ITIL Foundations Certified
IBM Certified Deployment Professional 
        --Tivoli Data Warehouse v1.2
        --Tivoli Enterprise Console v3.8 Network
Management

IBM Software Group - Tivoli Software
Research Triangle Park,  North Carolina.

"Evans, Bill" <Bill.Evans@hq.doe.gov>
Sent by: owner-nv-l@lists.us.ibm.com

03/16/2005 10:49 AM

Please respond to
nv-l

To	nv-l@lists.us.ibm.com
cc
Subject	[nv-l] Problems with SNMP monitoring

I'm having a problem with the migration of
NetView to a new machine.

This is a new SUSE SLES 9 installation of NetView 7.1.4 FP 2 on a Dell 1750 with manual transfer of seed, community strings, hosts, location.conf and other configuration data.  We are in a "test" mode.  It is using net-SNMP.  Our old system is a SUN with NV 7.1.3 and current fixpacks.  It uses the SUN SNMP.   We staged the bring up of the new machine to verify it's capacity and clean up the messy existing configuration.  Our first pass was to bring across the routers, then the switches, then the servers we monitor and finally any local extensions.  We're there with the full NetView device load.  

The area which is giving us problems is the SNMP management of Routers.  This includes 15 core network routers, 15 MAN routers and 37 Wide Area Network routers.  Core Routers are Cisco 6000 and 7000 models. WAN routers are  Cisco 3800 series.  MAN routers are all over the place from Cisco 2500 through 7500 models.      

The OLD machine is giving us fits with what appears to be dropped SNMP responses.  The particular ones giving trouble are the WAN devices although the loss of responses also hits the core routers on occasion.  It would appear that the SUN SNMP subsystem is swallowing some responses (randomly but tending toward the last ones received for the devices affected).  This began after we added a hundred or so HSRP interfaces to our core configuration.  These false alarms upset our management team and we're trying to address it by moving to a new box.

The new box works well (most of the time) for these devices.  When it is working it gives a reliable view of the state of the WAN routers. The "lost responses" are not a problem on the new machine.  Occasionally (about every 32 hours for the past couple days) a portion of the WAN if not all of it goes critical with SNMP polling timeouts.  When it happens, all the affected routers fail at the same time.  Until reset manually they will not recover.  One or more core routers may also be hit.

·       PING will work to the devices on either loopback or active port address but the device state will return to Critical because the next SNMP poll will fail.

·       SNMP polling is in use because the router configuration has a delay defined on one port (backup circuit) which prevents successful ICMP polling.  

·       QuickTest and QuickTest Critical will NOT work after the initial failure.  The result is an SNMP timeout.  
·       Demand Poll will work.  This resets whatever is ailing and all works well for another day.  
·       During the Demand Poll there is often a significant pause (up to one minute) after we see the "Get CDP Cache entry" line and sometimes another when we see the "Get MPLS MIB" line.  

·       The other machine is having no problems with its SNMP polling except for the continuing false alarms.

As you can guess this 32 hour cycle slows debugging.  A couple days ago I did an SNMP Walk on the devices but I'm not sure if it worked or didn't.  Next time I get a failure I plan to dig into that issues.  Meanwhile I haven't been able to find anything on the archives of in the knowledge base which appears to be similar.

 

I don't feel I have enough to go on to open an incident yet and hope the "communal wisdom" may point me in the right direction.   My current hypothesis:

·       The problem has to be in the NetView at the new machine.  

Suggestions and comments are solicited.  

Bill Evans