John,
I think you should Identify the devices in your network that has a
large routing table. Once identified, you can force the router to
prematurely end the queries of the route table from the NMS. Here is an old
note from the NV-L archives. Hope this helps:
IP Simple Network Management Protocol (SNMP) Causes High
CPU Utilization
This Tech Note explains how to troubleshoot high CPU utilization in a
router due to the IP_SNMP process, which can result from a
network management station using an SNMP walk to "find" the topology of a
network, coupled with a large route table in a smaller
router. This problem can effect lower end Cisco routers.
If you notice high CPU utilization on a Cisco router, and you determine the
cause is isolated to the IP_SNMP process using the output of
the show process cpu command, check the output of the debug ip snmp command
to see if the SNMP queries are being made to the
router's entire route table. As always, take precaution when running a
debug over a production network due to the potential for
overwhelming the router.
Network Management stations query routers for their entire route table to
learn what other networks the router knows about. It uses this
information to find other routers, and query them about their knowledge of
networks around them. In this fashion, the management
station can learn the topology of the entire network. The route table is
stored in the router in a hashed format, more conducive to quick
route searches. However, SNMP requests for the route require the router to
first sort the table into sequential order due to the way in
which the router responds to queries. The management station sends requests
asking for the "next route entry" one entry at a time until
the router has sent the whole table. Every time the router sees a "get next
route" request, the router must determine the last route it sent,
then sort the entire table sequentially to find the next route to send in
sequence. This process is very CPU intensive, as the entire routing
table is sorted from top to bottom every time it gets one of these
requests, and it gets a request for each individual line in the route
table.
SNMP is a low priority process as far as the CPU scheduler is concerned, so
if another process requires CPU resources, it generally takes
priority. As such, while CPU spikes occur in this scenario, they shouldn't
effect performance at first. To avoid performance issues, force
the router to prematurely end the queries for the route table from the
network management system server. Configure the router to respond
with a "complete" message as soon as it receives the start of a request for
the route table, as follows:
snmp-server view cutdown internet included
snmp-server view cutdown ipRouteTable excluded
snmp-server view cutdown ipNetToMediaTable excluded
snmp-server view cutdown at excluded
snmp-server community public view cutdown RO
snmp-server community private view cutdown RW
This configuration blocks requests to retrieve the route table
(ipRouteTable) and the Address Resolution Protocol (ARP) table
(ipNetToMediaTable), but allows all other requests through. Older versions
of Cisco IOS® Software don't recognize the MIB object
ipRouteTable, so the following configuration should be used instead.
snmp-server view cutdown internet included
snmp-server view cutdown ip.21 excluded
snmp-server view cutdown ip.22 excluded
snmp-server view cutdown at excluded
snmp-server community public view cutdown RO
snmp-server community private view cutdown RW
In both of these examples, you can substitute your own community strings.
The outcome of these configuration changes is that the router doesn't
return either the ARP table or the IP route table when queried. This
prevents SNMP network discoveries from creating CPU spikes on the router in
question, but it also removes a degree of manageability
from the router.
As a final note, routers that previously were not exhibiting this problem
may begin to do so if there is a change in their routing table. The
amount of cycles required to reply to the IP route table requests is a
function of the number of routes in the routing table. If the number
of routes increases, CPU utilization increases as well.
"John Sobrinho"
<johnsobrinho@rogers.co T
m> To:
<nv-l@lists.us.ibm.com>
cc:
Sent by:
owner-nv-l@lists.us.ibm bcc:
.com Subject:
RE: [nv-l] Newbie Question -
Auto Map re-gen and Net MON
03/14/2004 08:32 PM
Please respond to nv-l
How about splitting the seed file and add sections at time until we
complete the discovery at a rate of 250 devices / cycle ?. Initial cylce do
the map regen, and subseuent cycles jut stop start net mon ?
-----Original Message-----
From: owner-nv-l@lists.us.ibm.com
[mailto:owner-nv-l@lists.us.ibm.com]On Behalf Of Leslie Clark
Sent: Sunday, March 14, 2004 8:18 PM
To: nv-l@lists.us.ibm.com
Subject: RE: [nv-l] Newbie Question - Auto Map re-gen and Net MON
We should get some input from others on this. I would never say
never. There have been times when certain Cisco routers were badly
affected by excessive SNMP requests. Core switches should be a
different case, I would think. My belief is that if you don't have
the network capacity for network management traffic, then you don't
have enough network capacity. In that case, you could discover
slowly.
Others may be able to warn you about special vulnerabilities of
specific devices.
Cordially,
Leslie A. Clark
IBM Global Services - Systems Mgmt & Networking
Detroit
"John Sobrinho"
<johnsobrinho@rogers.com>
Sent by:
owner-nv-l@lists.us.ibm.com To
<nv-l@lists.us.ibm.com>
cc
03/14/2004 06:59 PM
Subject
RE: [nv-l] Newbie Question -
Please respond to Auto Map re-gen and Net MON
nv-l
Thank you Leslie..
Yes, I do have the need to do a regen.
Is my worry about any excessive traffic bringing a device down
unwarranted ? The reason why I ask, and am a bit afraid is that my
predecessor attempted to do a full discovery and brought down a core
switch in one of the buildings. Hence I am more careful with my
discovery needs, and setting them up accordingly.
We have an off site drp server, but it uses is strictly as trap
acceptor and sending alerts. We will not support mapping and db
functions in drp.
Hence my fear...
Thanks again,
John
-----Original Message-----
From: owner-nv-l@lists.us.ibm.com
[mailto:owner-nv-l@lists.us.ibm.com]On Behalf Of Leslie Clark
Sent: Sunday, March 14, 2004 2:52 PM
To: nv-l@lists.us.ibm.com
Subject: Re: [nv-l] Newbie Question - Auto Map re-gen and Net MON
Otherwise, if you want to do a new discovery, the re-gen will stop
netmon and everything else, and go out and rediscover everything,
depending on your discovery settings.
Cordially,
Leslie A. Clark
IBM Global Services - Systems Mgmt & Networking
Detroit
"John Sobrinho"
<johnsobrinho@rogers.com>
Sent by:
owner-nv-l@lists.us.ibm.com To
<nv-l@lists.us.ibm.com>
cc
03/14/2004 02:23 PM
Subject
[nv-l] Newbie Question - Auto
Please respond to Map re-gen and Net MON
nv-l
Hello All...
I have a newbie question.
I'm in a situation where one of the net view servers has been
neglected for some time and the map and db is pretty much useless. I
have cleaned up the seed file, and removed as many entries from the
host file as I can as we would try to rely on DNS for resolution as
well I have created a Location conf file based on our ospf areas.
What I'd like to accomplish is zero out the db's and create a new map
and have fresh db's (start from scratch).
My question is , do I only need re-gen the map from the admin panel,
to ensure everything is discovered, or do I need to recycle netmon,
then do a map regen ?
My concern is that we have well over 1000 entries in the seed file
(loop back address of various L2 / L3 devices) and am concerned that
the traffic may bring down the switch or router, we are 7/24 hr shop
so no chance of doing this during maint. windows. Netview is
configured to do only local discovery and seed file, no wild card
ranges defined.
The box is NV 7.1. AIX 4.3.3 on multiple 100 MB nics.
Thanks in advance
John Sobrinho
|