When hazelcast clustering is enabled, a plugin can create it's own Cache, using something like:
final String cacheName = "aaa.test-cache";
final Cache<String, String> cache = CacheFactory.createCache(cacheName);
This appears to work because it is possible to dynamically configure a Hazelcast IMap (which is what backs the Openfire Cache when clustered) provided the IMap has not already been created.
However, as soon as a second node joins the cluster, the second node will create an IMap instance for each one existing on the first node. As there is no explicit config for "aaa.test-cache" it gets created on the second node with default values (set to infinite in hazelcast-cache-config.xml) - and this config is then reflected back on the first node. As a result, as soon as the second node joins, entries stop expiring on all nodes.
Potential solution include:
1. Remember to manually edit the single hazelcast-cache-config.xml file on each server every time a plugin is installed that uses a Cache. In which case this should be closed as "Won't fix".
Pros: No development effort
Cons: Manual admin process required, no doubt prone to error
2. When ClusteredCache puts an entry in the IMap, it could use map.put(key, object, maxLifetime, TimeUnit.Milliseconds) to explicitly set the expiry time on the entry, instead of the cache.
Pros: Trivial to implement (already done to see if it works!)
Cons: Doesn't work for the putAll(Map) method which has no equivalent. Doesn't work for the setMaxSizeProperty() property.
3. HZ config files support an <import resource="path/to/file.xml"> directive. The Hazelcast plugin could find any plugin specific cache configurations (cf. the coherence clustering plugin with the cache-config.xml file), and programmatically add these files to the cluster config before clustering starts.
Pros: Works in all cases
Cons: Non-trivial development effort.
4. Upgrade to HZ 3.9, which supports dynamic configuration
Pros: Works in all cases with little development effort (depending on how much the API has changed).
Cons: HZ 3.9 is fairly new (only two days old at the time of writing!) - some time to allow others to find any bugs might be beneficial.
I'm open to any other suggestions, thoughts, etc. In the meantime, I'm going to implement option 2 locally as a short-term work-around.
[EDIT #2: Manually number solutions as markup numbering seems to be failing]
Note; fix #2 was implemented in plugin v. 2.2.4 - however the max size is still not properly honoured.