windows - Openfire Cluster Hazelcast Plugin Issues -

windows server 2003r2/2008r2/2012, openfire 3.8.1, hazelcast 1.0.4, mysql 5.5.30-ndb-7.2.12-cluster-gpl-log

we've set 5 servers in openfire cluster. each of them in different subnet, subnets located in different cities , interconnected each other through vpn routers (2-8 mbps):

192.168.0.1 - node0 192.168.1.1 - node1 192.168.2.1 - node2 192.168.3.1 - node3 192.168.4.1 - node4 

openfire configured use mysql database successfully replicating master node0 slave nodes (each node uses it's own local database server, functioning slave).

in openfire web admin > server manager > clustering able see cluster nodes.

openfire custom settings hazelcast:

hazelcast.max.execution.seconds - 30 hazelcast.startup.delay.seconds - 3 hazelcast.startup.retry.count - 3 hazelcast.startup.retry.seconds - 10 

hazelcast config node0 (similar on other nodes except interface section) (%programfiles%\openfire\plugins\hazelcast\classes\hazelcast-cache-config.xml):

<join>   <multicast enabled="false" />    <tcp-ip enabled="true">     <hostname>192.168.0.1:5701</hostname>      <hostname>192.168.1.1:5701</hostname>      <hostname>192.168.2.1:5701</hostname>      <hostname>192.168.3.1:5701</hostname>      <hostname>192.168.4.1:5701</hostname>   </tcp-ip>   <aws enabled="false" />  </join> <interfaces enabled="true">   <interface>192.168.0.1</interface>  </interfaces> 

these settings changed default ones.

the problem xmpp clients authorizing long, 3-4 minutes, after authorization other users in roster inactive 5-7 minutes, during time logged in user in openfire web admin > sessions marked offline. after user able see other logged in users active, messages not delivered, or delivered after 5-10 minutes or after few openfire restarts...

we appreciate help. spent 5 days trying set monster, , out of ideas... :(

thanks lot in advance!

upd 1: installed openfire 3.8.2 alpha hazelcast 2.5.1 build 20130427 same problem

upd 2: tried starting cluster on 2 servers in same city, separated 1-2 hops @ 1-5ms ping. works perfectly! stopped 1 of servers , started 1 in city (3-4 hops @ 80-100 ms ping) problem occured again... slow authorizations, logged off users in roster, messages not delivered on time etc.

upd 3: installed openfire 3.8.2 without jre, , java sdk 1.70_25.

here jmx screenshots:

node 0:

node 1:

red line first client connection (after openfire restart). tested on 2 users. same thing... first user (node0) connected instantly, second user (node1) spent 5 seconds on connection. rosters have been showing offline users on both sides 20-30 seconds, online users start appearing in them. first user sends message second user. second user waits 20 seconds, receives first message. reply , other messages transfered instantly.

upd 4:

durring diggin through jconsole "threads" tab we've discovered these various states:

for example hz.openfire.cached.thread-3:

waiting on java.util.concurrent.synchronousqueue$transferstack@8a5325 total blocked: 0  total waited: 449 

maybe help... don't know for.

thanks!

[update] note per hazelcast documentation - wan replication supported in enterprise version only, not in community version shipped openfire. must obtain enterprise license key hazelcast if use feature.

you may opt setup multiple lan-based openfire clusters , federate them using s2s integration across separate xmpp domains. preferred approach scaling openfire large user base.

[original post follows]

my guess longer network latency in remote cluster configuration might tying hazelcast executor threads (for queries , events). of these events , queries invoked synchronously within openfire cluster. try tuning following properties:

hazelcast.executor.query.thread.count (default: 8) hazelcast.executor.event.thread.count (default: 16) 

i start setting these values 40/80 (5x) respectively see if there improvement in overall application responsiveness, , potentially higher based on expected load. additional hazelcast settings (including other thread pools) plus instructions adding these properties configuration xml can found here:

hazelcast configuration properties

hope helps ... , luck!