When admin tries to remove JmxWeb plugin, Openfire can't cleanly remove all related files. One has to manually remove ijmxweb folder from plugins folder. Restarting Openfire also fixes the problem.
Console is logging: Error unloading plugin 'jmxweb'. Will attempt again momentarily.
Note; this is actually a bug in Jetty - https://github.com/eclipse/jetty.project/issues/1425 - and affects any plugin that has a webapp using the client/bosh port 7070.