Split this issue in smaller issues for a better handling.
Now after spliting the issue only the createMUCService patch is related to this issue. The validation of the service name is done by creating a new JID before the service is created. The maybe thrown IllegalArgumentException and the AlreadyExistsException is handled by the admin console.
The proposed fix has one drawback: the validation of the service name of a MUC service is not done in the MUC service implementation (implementations of org.jivesoftware.openfire.muc.MultiUserChatService, where it belongs.
Guenther explained me offline that modifying the patch is not as straight-forward as it looks. Currently, the service name is persisted before it gets used in the constructor of MultiUserChatServiceImp, which in turn (indirectly) depends on the values being persisted.
We might be able to fix this by having the MUC component being initialized by the initialize-method provided by the Component interface, instead of through the constructor. This method is called when the MUC service is added to the Component manager. Moving code around like this would not only solve our original problem, but also clean up some of the MUC code.
The new patch verifies the service name on the creation of a MultiUserChatService Object. Therefore the initialization is modified to allow the creation without database access. Also a separate error message is added for already existing services.
Guenther's latest patch is cleaner than the previous one, and makes sure that every attempt to create a MUC service with an illegal name fails. The patched code works flawlessly for me, locally.