We're updating the issue view to help you get more done. 

Console error in session view when a cluster member goes offline

Description

Per email from Guus:

We quite often upgrade the domain at igniterealtime.org with a new Openfire build. Even when no cluster-related code has changed, we frequently get errors - notably on the admin panel page that displays the current user sessions. Stack traces will appear there, complaining about the unavailability of a particular cluster node identified by some byte array. The issue disappears after restarting both cluster nodes, while making sure that at some point in time all nodes are shut down.

Environment

None

Acceptance Test - Entry

None

Activity

Show:
Tom Evans
November 20, 2012, 7:28 PM

This is an example of the error as reported:

java.lang.IllegalStateException: Requested node [B@3fa6b386 not found in cluster
at com.jivesoftware.util.cache.ClusteredCacheFactory.doSynchronousClusterTask(ClusteredCacheFactory.java:328)
at org.jivesoftware.util.cache.CacheFactory.doSynchronousClusterTask(CacheFactory.java:566)
at com.jivesoftware.openfire.session.RemoteSession.doSynchronousClusterTask(RemoteSession.java:173)
at com.jivesoftware.openfire.session.RemoteSession.isSecure(RemoteSession.java:131)
at org.jivesoftware.openfire.admin.session_002dsummary_jsp._jspService(session_002dsummary_jsp.java:362)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:547)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1359)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)
at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:74)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)
at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:50)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)
at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:78)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)
at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:164)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1330)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:941)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:409)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:875)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:441)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:919)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:582)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
at java.lang.Thread.run(Thread.java:662)

This can be reproduced by loading the session page from a peer member after a cluster member has been shut down.

Tom Evans
November 20, 2012, 11:25 PM
Edited

Validate sessions before displaying in session UI. (SVN 13369/13370)

Tom Evans
January 9, 2013, 10:18 PM

Initial fix covered the symptoms but not the root of the problem. Additional changes pending.

Tom Evans
January 9, 2013, 10:29 PM

Per 13392: Improved exception handling and error recovery in core clustering classes; retooled cache configuration to prevent eviction of core components; updated versions and documentation for clustering and hazelcast plugins

Tom Evans
January 9, 2013, 10:31 PM

NOTE: Need to update both Openfire core and Hazelcast plugin (1.0.2) to pick up this fix.

Assignee

Tom Evans

Reporter

Tom Evans

Expected Effort

None

Ignite Forum URL

None

Components

Fix versions

Affects versions

Priority

Critical
Configure