Major performance hit with MINA 2.0.9 vs 2.0.7
Description
Environment
Activity
Daryl Herzmann June 19, 2015 at 12:38 PM
(06/18/2015 10:37:06 PM) Andi: Alright so the performance tests from today are looking pretty good. 3.10.1 with Tom's changes to revert mina back to 2.0.7 are on par with the performance of 3.9.3.
(06/18/2015 10:38:06 PM) Andi: The test we ran today was to get about 19k users logged in, have them IM with each other as well as create and join some of about 250 rooms.
(06/18/2015 10:38:38 PM) Andi: Some of these rooms ended up with 200 to 350 users, all of which are posting messages. Not at crazy rates but somewhat realistic scenarios
(06/18/2015 10:39:00 PM) Andi: So overall a pretty decent load of connected users, IM messages as well as MUC activity going on.
(06/18/2015 10:41:09 PM) Andi: Also the earlier thing I mentioned about higher mem usage of 3.10.1 is a non issue. while at the end after the test, 3.10.1 ended up with more 'used' memory, most of that was unreachable objects and just didn't get garbage collected yet. Here's the heapdump MAT analysis of that: https://www.dropbox.com/s/mgy6yd23rn9fo6z/3.10.1_heapdump_20150618_high_mem_usage.zip?dl=0
(06/18/2015 10:41:58 PM) Andi: and here are the 4 Tsung reports for the 2 runs of the test against 3.9.3 and 2 runs of the test against 3.10.1 with Tom's changes. Also Openfire was not restarted between the 2 3.10.1 runs.
(06/18/2015 10:42:16 PM) Andi: https://www.dropbox.com/s/uj01y91ulfssoqg/muc20k-3.9.3-20150618-1005.zip?dl=0
(06/18/2015 10:42:29 PM) Andi: https://www.dropbox.com/s/gffzu5ak9tlsgop/muc20k-3.9.3-20150618-1126.zip?dl=0
(06/18/2015 10:42:42 PM) Andi: https://www.dropbox.com/s/7wku413ddq09fnh/muc20k-3.10.2-20150618-1748.zip?dl=0
(06/18/2015 10:42:55 PM) Andi: https://www.dropbox.com/s/vikfr7lvbztqvu4/muc20k-3.10.2-20150618-1936.zip?dl=0
Tom Evans June 17, 2015 at 10:58 PM
PR submitted against master (3.11.0), but can also be merged into 3.10 branch as needed to support a 3.10.2 patch release.
Tom Evans June 17, 2015 at 10:57 PM
PR submitted: https://github.com/igniterealtime/Openfire/pull/241
Tom Evans June 17, 2015 at 9:17 PM
I'll grab this one. Looks like we need to (at least) revert this commit:
https://github.com/igniterealtime/Openfire/commit/89c06a88d75e15d36ce7f6ace88bbf5c71fb08df
and maybe several others related to NIOConnection.
Andi has been doing some awesome performance testing of Openfire and finds that there is a major regression of scalability from Openfire 3.9.3 to 3.10.1. Here are some Tsung reports
3.9.3 reports: https://www.dropbox.com/s/blqybi8sw6x7tei/15ksimul3.9.3-20150617-0918.zip?dl=0
3.10.1 reports: https://www.dropbox.com/s/ri8kpxihe1tlre7/15ksimul3.10.1-20150617-1051.zip?dl=0
Andi then did a test with simply replacing MINA in 3.10.1 from version 2.0.9 to 2.0.7
Here's the test result for 3.10.1 with mina 2.0.7. Very similar to the 3.9.3 performance. Also got to around 15k users with memory being stable around 300-400MB. https://www.dropbox.com/s/23ucnvhtvqze6m4/15ksimul3.10.1-mina2.0.7-20150617-1252.zip?dl=0
This MINA Jira issue may be related
https://issues.apache.org/jira/browse/DIRMINA-934
It may be time to consider downgrading Openfire to MINA 2.0.7