[Top] [All Lists]

Re: [NV-L] Application reached maximum number of outstanding events,

To: James Shanks <jshanks@us.ibm.com>, Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>
Subject: Re: [NV-L] Application reached maximum number of outstanding events,
From: Larry Fagan <larrytechie@yahoo.com>
Date: Mon, 19 Mar 2007 10:00:09 -0700 (PDT)
Delivery-date: Mon, 19 Mar 2007 17:10:35 +0000
Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=PFtkrZe0dgL1vIMryj6SAYkYe9vEHizSKzRssPi2I57/an4T46W+B9W8NEjJC2oPRkkV239dD9uHIvHJUrcWDQCllDoNu4crFoJH1reVY4gqVvBsJjE3nOoA0uyxbGpQhvez8/vnlbpz7JAFdAQL+xOVJvjVTdY6fVG1jFryJec=;
Envelope-to: nv-l-archive@lists.skills-1st.co.uk
In-reply-to: <OF8D02912C.D5315929-ON852572A1.0005A95B-852572A1.000829F8@us.ibm.com>
List-help: <mailto:nv-l-request@lists.ca.ibm.com?subject=help>
List-id: Tivoli NetView Discussions <nv-l.lists.ca.ibm.com>
List-post: <mailto:nv-l@lists.ca.ibm.com>
List-subscribe: <http://lists.ca.ibm.com/mailman/listinfo/nv-l>, <mailto:nv-l-request@lists.ca.ibm.com?subject=subscribe>
List-unsubscribe: <http://lists.ca.ibm.com/mailman/listinfo/nv-l>, <mailto:nv-l-request@lists.ca.ibm.com?subject=unsubscribe>
Reply-to: larrytechie@yahoo.com, Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>
Sender: nv-l-bounces@lists.ca.ibm.com
thanks for looking into the issue.. Below are answers to the questions and some more info..
This issue happenned before about 6 months  or so back and IBM support asked me to increase the application queque log size from 2000 to 10000 and the issue vanished. So this started again last week and i again increased from 10000 to 15000 and has not happenned again. I'm not sure if that's the issue...
-When i check nvstatus, all are up and running
-Yes, popup "nvcorrd is running" is displayed when NetView console is open when this issue happens
-Yes, we keep adding router or switches or server as trap destination as our Netview server every other day. I don't see any trap storms. If so , theh why does'nt happen evry often?
-Did'nt add any rule set off late. Does'nt look like rule set issue.
- no changes to ESE. automation, TEC rule set

Do you think,by upgrading to latest version, are the netmon, trapd are better capable for performance.
As is said, i have 7.1.3 FP3. Which is the most stable current version to upgrade to?
Thanks in advance as usual.

James Shanks <jshanks@us.ibm.com> wrote:

More information please,  Larry.  These sorts of problems seldom arise out of the blue.

That message means just what it says, some unnamed application disconnected from trapd.  If that application is the event command, then this message is normal.  But otherwise it points to a processing/ performance problem somewhere else. When an application registers with trapd to receive traps, trapd assigns it an in-storage queue,  If that application becomes so busy that it cannot process the traps that trapd passes it, and the queue becomes full, trapd disconnects that application in order to save himself, and issues a message similar to the one you have indicated.  So the question becomes  (1) who has exited and (2) why.  Usually the answer is that the process which exited has been given too much work to do and cannot keep up.  Sometimes this is the result of a code bug, sometimes it is the result of a configuration change.    So what does ps -ef or ovstatus tell you?  Did any daemon go away?  Did you get a pop-up or an error message in nvevents or ipmap?

Even if you didn't, then  question to answer is what had changed just before  you have started having this problem?  Did you add a new ruleset?  Change the TEC ruleset?   Ad something to ESE,automation?  Configure a bunch of new routers to send traps to trapd?   If the process which exited is nvcorrd, then the culprit is most likely a badly-coded user ruleset.  If it is nvserverd, then sometimes the issue is an new user trying to connect over a slow link.  How long have you had the application queue size set to 15000?  Why did you change it?  That will work if the problem is that you have periodic trap storms, but other wise not.  In a storm, trapd has to suspend giving incoming traps to the connected apps while he just reads the incoming stuff off his socket.  He has to do that so that he doesn't lose any.  But once the storm subsides, he processes the accumulated traps as rapidly as possible and puts them on each connected application's queue.  The result is that each of them goes from idle to having a large amount of work to do, and thus they need a big queue to hold it all, while they work. But if the problem is not caused by a trap storm, then increasing the queue size to 15000 may just mask the problem.  It just delays the inevitable disconnection.

So what else is new in your environment?  If you tell me nothing, then all I can recommend is that you apply more current code and call Support.  The last fixpack for 7.2.3 was Version 4 and it's over a year old by now.  I'm not even sure whether the level you have would allow you to create an nvserverd .log and trace what's being sent to TEC.

James Shanks
Level 3 Support  for Tivoli NetView for UNIX and Windows
Network Availability Management
Network Management - Development
Tivoli Software, IBM Corp

Larry Fagan <larrytechie@yahoo.com>
Sent by: nv-l-bounces@lists.ca.ibm.com
03/16/2007 05:27 PM
Please respond to
larrytechie@yahoo.com; Please respond to
Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>

Tivoli NetView Discussions <nv-l@lists.ca.ibm.com>

[NV-L] Application reached maximum number of outstanding events,

Hi All,
I'm running into an issue here. I get often the above message and no events show up on the NetView control desk neither are forwarded to TEC. I have set the trapd connected application queue size as 15000. I have 7.1.3 FP3 on AIX.
Any ideas what can be done? Is there a trap defined for the above message and can be forwarded to TEC or send alerts so that i can take action upon.
Please advice?

We won't tell. Get more on shows you hate to love
(and love to hate):
Yahoo! TV's Guilty Pleasures list._______________________________________________
NV-L mailing list
http://lists.ca.ibm.com/mailman/listinfo/nv-l (Browser access limited to internal IBM'ers only)

Expecting? Get great news right away with email Auto-Check.
Try the Yahoo! Mail Beta.
NV-L mailing list
http://lists.ca.ibm.com/mailman/listinfo/nv-l (Browser access limited to 
internal IBM'ers only)
<Prev in Thread] Current Thread [Next in Thread>

Archive operated by Skills 1st Ltd

See also: The NetView Web