Re: [NV-L] Application reached maximum number of outstanding events,

[James Shanks <jshanks@us.ibm.com>]

More information please,  Larry.
 These sorts of problems seldom arise out of the blue.

That message means just what it says,
some unnamed application disconnected from trapd.  If that application
is the event command, then this message is normal.  But otherwise
it points to a processing/ performance problem somewhere else. When an
application registers with trapd to receive traps, trapd assigns it an
in-storage queue,  If that application becomes so busy that it cannot
process the traps that trapd passes it, and the queue becomes full, trapd
disconnects that application in order to save himself, and issues a message
similar to the one you have indicated.  So the question becomes  (1)
who has exited and (2) why.  Usually the answer is that the process
which exited has been given too much work to do and cannot keep up.  Sometimes
this is the result of a code bug, sometimes it is the result of a configuration
change.    So what does ps -ef or ovstatus tell you?  Did
any daemon go away?  Did you get a pop-up or an error message in nvevents
or ipmap?

Even if you didn't, then  question
to answer is what had changed just before  you have started having
this problem?  Did you add a new ruleset?  Change the TEC ruleset?
  Ad something to ESE,automation?  Configure a bunch of new routers
to send traps to trapd?   If the process which exited is nvcorrd,
then the culprit is most likely a badly-coded user ruleset.  If it
is nvserverd, then sometimes the issue is an new user trying to connect
over a slow link.  How long have you had the application queue size
set to 15000?  Why did you change it?  That will work if the
problem is that you have periodic trap storms, but other wise not.  In
a storm, trapd has to suspend giving incoming traps to the connected apps
while he just reads the incoming stuff off his socket.  He has to
do that so that he doesn't lose any.  But once the storm subsides,
he processes the accumulated traps as rapidly as possible and puts them
on each connected application's queue.  The result is that each of
them goes from idle to having a large amount of work to do, and thus they
need a big queue to hold it all, while they work. But if the problem is
not caused by a trap storm, then increasing the queue size to 15000 may
just mask the problem.  It just delays the inevitable disconnection.

So what else is new in your environment?
 If you tell me nothing, then all I can recommend is that you apply
more current code and call Support.  The last fixpack for 7.2.3 was
Version 4 and it's over a year old by now.  I'm not even sure whether
the level you have would allow you to create an nvserverd .log and trace
what's being sent to TEC.

James Shanks
Level 3 Support  for Tivoli NetView for UNIX and Windows
Network Availability Management
Network Management - Development
Tivoli Software, IBM Corp

Larry Fagan <larrytechie@yahoo.com>
Sent by: nv-l-bounces@lists.ca.ibm.com

03/16/2007 05:27 PM

Please respond to
larrytechie@yahoo.com; Please respond to
Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>

To	Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>
cc
Subject	[NV-L] Application reached maximum number of outstanding events,

Hi All,
I'm running into an issue here. I get often the above
message and no events show up on the NetView control desk neither are forwarded
to TEC. I have set the trapd connected application queue size as 15000.
I have 7.1.3 FP3 on AIX. 
Any ideas what can be done? Is there a trap defined for
the above message and can be forwarded to TEC or send alerts so that i
can take action upon.
Please advice?
Thanks,
Larry

 

---

We won't tell. Get more on shows
you hate to love
(and love to hate): Yahoo!
TV's Guilty Pleasures list._______________________________________________
NV-L mailing list
NV-L@lists.ca.ibm.com
Unsubscribe:NV-L-leave@lists.ca.ibm.com
http://lists.ca.ibm.com/mailman/listinfo/nv-l (Browser access limited to
internal IBM'ers only)

_______________________________________________
NV-L mailing list
NV-L@lists.ca.ibm.com
Unsubscribe:NV-L-leave@lists.ca.ibm.com
http://lists.ca.ibm.com/mailman/listinfo/nv-l (Browser access limited to 
internal IBM'ers only)