This page last changed on Sep 15, 2011 by edawson.
Please be aware that Confluence clustered is not available for version 4.0 yet. It will be forthcoming in a minor release of Confluence following the release of version 4.0.

 

Confluence on Virtualised Environments


Atlassian officially supports non-clustered installations of Confluence 3.0 and later on VMware. Although possible, we do not recommend (nor support) running versions of Confluence prior to 3.0 on VMware, since Confluence 3.0 resolved many performance issues that were present in earlier versions. Be aware that we also do not support clustered installations of Confluence on VMware. Please comment or vote on the feature request at CONF-19559.

Overview of clustering documentation

Refer to the overview of Confluence clustering.

This page covers troubleshooting for the Clustered Edition. If you're experiencing Cluster Panic messages in a Standard Edition, visit the Knowledge Base article 'Database is being updated by an instance which is not part of the current cluster' Error Message.

On this page:

Symptoms

Below is a list of potential problems with a Confluence cluster, and their likely solutions. The solutions are listed below.

Problem

Likely solutions

Database is being updated by an instance which is not part of the current cluster errors on a stand-alone

'Database is being updated by an instance which is not part of the current cluster' Error Message

Database is being updated by an instance which is not part of the current cluster errors on a cluster

Cluster Troubleshooting, Cluster Troubleshooting

Cannot assign requested address on startup, featuring an IPv6 address

Cluster Troubleshooting

Error in log: The interface is not suitable for multicast communication

Cluster Troubleshooting, Cluster Troubleshooting

Multicast being sent, but not received (detectable with Multicast Test)

Cluster Troubleshooting, Cluster Troubleshooting, Cluster Troubleshooting

Any issue not covered here

Cluster Troubleshooting

Confluence cluster debugging tools

There is an umbrella issue opened for all cluster debugging tools here

It includes the tools listed below.

Multicast

  • Which multicast address?

The multicast address and port used by Confluence can be found on the Cluster Administration page, or in confluence.cfg.xml in the Confluence home directory.

  • Multicast address generation.

Confluence uses a hashing algorithm to take the inputted name during setup and it is then turned into a multicast address stored in the config file. Thus, once the initial setup is completed, Confluence will use the address this is the reason why user can change the address if needed, without actually changing the name. Consequently the additional nodes using the same multicast address specified in the config file are able to join the cluster.

Each node has a multicast address configured in the confluence-cfg.xml file

name="confluence.cluster.address">xxx.xx.xxx.xxx</property>

A warning message is displayed when an user changes the address from the one that Confluence has generated by the hashing of the name. There is no way of eliminating the message any other way other than by returning the address to the one that matches the cluster name. Purpose of the warning message is to remind the user that the address has been changed - as it is not the hashed version any longer - consequently the node can not join the cluster just by using the name. It is also necessary to provide the correct address as well.

Mapping interface to IP address.

To ensure that the interface name is mapped correctly, the following tool can be used. It shows the mapping of the interface name to the IP address.

C:\>java -jar list-interfaces.jar
interfaces.size() = 4
networkInterface[0] = name:lo (MS TCP Loopback interface) index: 1 addresses:
/127.0.0.1;

networkInterface[1] = name:eth0 (VMware Virtual Ethernet Adapter for VMnet8) index: 2 addresses:
/192.168.133.1;

networkInterface[2] = name:eth1 (VMware Virtual Ethernet Adapter for VMnet1) index: 3 addresses:
/192.168.68.1;

networkInterface[3] = name:eth2 (Broadcom NetXtreme 57xx Gigabit Controller - Packet Scheduler Miniport) index: 4 addresses:
/192.168.0.101;

Debugging tools

Listed below are some debugging tools that help determine what the status of the multicast traffic is:

Tool

Information provided

netstat -gn

Lists multicast groups. Does not work on Mac OS X.

netstat -rn

Lists system routing table.

Multicast Test

Coherence tool for testing multicast traffic from one node to another.

tcpdump -i interface

Captures network traffic on the given interface. Most useful on an interface that only receives cluster traffic.

Add multicast route

Multicast networking requirements vary across operating systems. Some operating systems require little configuration, while some require the multicast address to be explicitly added to a network interface before Confluence can use it.

If the Multicast Test tool shows that multicast traffic can't be sent or received correctly, adding a route for multicast traffic on the correct interface will often fix the problem. The example below is for a Ubuntu Linux system:

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

To support multiple applications using multicast on different interfaces, you may need to specify a route specific to the Confluence multicast address.

Check firewall

Ensure your firewall allows UDP traffic on the multicast address and port used by Confluence.

Prefer IPv4

There's a known issue with IPv6, especially on Linux.

The fix is to add -Djava.net.preferIPv4Stack=true to JAVA_OPTS. This tells the JVM to try binding an IPv4 address first, and resort to IPv6 only if that fails.
Note: A more radical approach is to add NETWORKING_IPV6=no to /etc/sysconfig/network, yet probably should be left for a later consideration on a production machine.

Change multicast interface

Confluence might have selected the incorrect interface for multicast traffic, which means it cannot connect to other nodes in the cluster. To override the interface used for multicast traffic after initial setup, edit confluence.cfg.xml in the Confluence home directory and add a property (or change the existing one) to select your desired network interface. For example to tell Confluence to use eth1:

<property name="confluence.cluster.interface">eth1</property>

Increase multicast TTL

The multicast time-to-live (TTL) specifies how many hops a multicast packet should be allowed to travel before it is discarded by a router. It should be set to the number of routers in between your clustered nodes: 0 if both are on the same machine, 1 if on two different machines linked by a switch or cable, 2 if on two different machines with one intermediate router, and so on.

Create a file in the Confluence home directory called tangosol-coherence-override.xml. Add the following to it, setting the TTL value appropriately (1 is the default):

<?xml version='1.0'?>
<coherence>
<cluster-config>
  <multicast-listener>
    <time-to-live system-property='tangosol.coherence.ttl'>1</time-to-live>
  </multicast-listener>
</cluster-config>
</coherence>

Alternatively, simply start Confluence with the system property: -Dtangosol.coherence.ttl=1. Again, 1 is the default value, and you should change it to something appropriate to your network topology.

Check intermediate routers

Advanced switches and routers have the ability to understand multicast traffic, and route it appropriately. Unfortunately sometimes this functionality doesn't work correctly with the multicast management information (IGMP) published by the operating system running Confluence.

If multicast traffic is problematic, try disabling advanced multicast features on switches and routers in between the clustered nodes. These features can prevent multicast traffic being transmitted by certain operating systems.

For best results, use the simplest network topology possible for the cluster traffic between the nodes. For two nodes, that means a single network cable. For larger numbers, try using a single high-quality switch.

Advanced Tangosol configuration

If the solution to your problem involves changes to the Tangosol configuration, these changes should not be made to the Confluence configuration in confluence/WEB-INF/classes/. Instead, to ensure your configuration survives upgrades, make your changes via:

  • Tangosol system properties
  • creating a tangosol-coherence-override.xml file in the Confluence home directory.

Examples of making these changes are shown in the increasing the TTL section.

Didn't find a solution?

Check Related Articles from the Confluence Knowledge Base

No content found for label(s) cluster.

Open JIRA Features and Bug Reports

JIRA Issues (56 issues)
Type Key Summary Assignee Reporter Priority Status Resolution Created Updated Due
Improvement CONF-9297 Confluence should be able to automatically recover from cluster panics Unassigned Gary Weaver Open Unresolved Aug 27, 2007 Mar 25, 2009
Improvement CONF-8716 Determine index mismatch in cluster and warn on cluster info page Unassigned Matt Ryall [Atlassian] Open Unresolved Jun 17, 2007 Oct 29, 2007
Improvement CONF-10980 Cluster debugging/troubleshooting tools Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 06, 2008 Mar 06, 2008
New Feature CONF-14948 Support failover NICs for cluster configuration... Unassigned Tony Atkins [Atlassian] Open Unresolved Mar 19, 2009 Mar 19, 2009
Improvement CONF-10635 Database logging of clustersafety access Unassigned James Fleming [Atlassian] Open Unresolved Feb 03, 2008 Feb 03, 2008
New Feature CONF-10953 Support unicast addressing in cluster when well-known-addresses WKA are defined Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 06, 2008 Aug 25, 2009
New Feature CONF-10977 Generate new Multicast address from a "new" cluster name Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 06, 2008 Sep 11, 2008
New Feature CONF-10979 List confluence cluster interface Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 06, 2008 Mar 06, 2008
Improvement CONF-10981 Check how many nodes/processes running in a cluster and their identity Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 06, 2008 Mar 06, 2008
Improvement CONF-11206 Confluence Clustered and JIRA trust delegation Unassigned Ivan Benko [Atlassian] Open Unresolved Mar 25, 2008 May 12, 2010
Improvement CONF-14338 Specify an arbitrary multicast port for a cluster Unassigned James Fleming [Atlassian] Open Unresolved Jan 30, 2009 Jan 30, 2009
Improvement CONF-9712 Plugins which don't work in a cluster shouldn't look like an error Unassigned Gary Weaver Open Unresolved Oct 15, 2007 Oct 16, 2007
New Feature CONF-19559 Provide support for Confluence clustered in a virtualized environment... Unassigned Tony Atkins [Atlassian] Open Unresolved May 06, 2010 Apr 08, 2011
New Feature CONF-19626 Support more than four clustered nodes Unassigned Tony Atkins [Atlassian] Open Unresolved May 12, 2010 May 16, 2010
Improvement CONF-20500 A cluster panic should not bring down other nodes Unassigned Partha Kamal [Atlassian] Open Unresolved Jul 30, 2010 Nov 08, 2010
Improvement CONF-20501 Consider upgrading coherence Unassigned Partha Kamal [Atlassian] Open Unresolved Jul 30, 2010 Jul 30, 2010
Improvement CONF-20974 Detect clusters on all network interfaces Unassigned Richard Atkins [Atlassian] Open Unresolved Oct 05, 2010 Oct 08, 2010
New Feature CONF-9335 In cluster, allow attachments to be stored on file system in network-shared directory Unassigned Jeremy Largman [Atlassian] Open Unresolved Aug 29, 2007 Sep 15, 2011
Improvement CONF-12689 Support Confluence cluster upgrades without an outage Unassigned Igor Minar Open Unresolved Aug 06, 2008 May 04, 2010
Improvement CONF-21670 Confluence should explicitly check for other confluence instances using the same home directory Unassigned Don Willis [Atlassian] Open Unresolved Jan 20, 2011 Jan 25, 2011
Improvement CONF-23223 Remove the option to store attachments on filesystem when using a cluster Unassigned Carlos Alberto Feijo Schedler [Atlassian] Open Unresolved Sep 14, 2011 Sep 15, 2011
Bug CONF-12287 Coherence cache fails while retrieving profile picture metadata (dashboard or view page shows UnexpectedRollbackException) Unassigned Matt Ryall [Atlassian] Critical Open Unresolved Jul 01, 2008 Apr 05, 2011
Bug CONF-14120 Hibernates UpdateTimestampsCache doesn't handle concurrent writes Unassigned Chris Kiehl [Atlassian] Critical Open Unresolved Jan 05, 2009 May 05, 2009
Bug CONF-8959 Attachment migration does not happen when upgrading to a clustered license Unassigned Nicholas Ilacqua [Atlassian] Critical Open Unresolved Jul 19, 2007 Oct 10, 2010
Task CONF-16794 Document new cluster distributions Giles Gaskell [Atlassian] Jeremy Largman [Atlassian] Critical Open Unresolved Sep 04, 2009 May 12, 2010
Bug CONF-9040 Authenticator (subclass of DefaultAuthenticator) can be called twice at almost exactly same time by 2 or more clustered servers Unassigned Gary Weaver Major Open Unresolved Jul 30, 2007 Nov 04, 2007
Bug CONF-9324 Lots of ObjectDeletedException's during cluster builds Unassigned Matthew Jensen [Atlassian] Major Open Unresolved Aug 28, 2007 May 12, 2010
Bug CONF-9594 ConditionalPropertySet's cannot be cached breaking cluster installations that delegate user management to JIRA Unassigned Dave Loeng [Atlassian] Major Open Unresolved Sep 28, 2007 Jul 02, 2009
Bug CONF-9813 Disable attachments migration to Filesystem in Cluster Unassigned Gurleen Anand [Atlassian] Major Open Unresolved Oct 24, 2007 Sep 04, 2011
Bug CONF-10323 Coherence Lock being held when it appears no thread should have the lock. Causes ConcurrentModificationException Unassigned Paul Curren [Atlassian] Major Open Unresolved Dec 26, 2007 Oct 25, 2010
Bug CONF-10325 Viewing the members of a group in a clustered environment works only on one node and not the other. Unassigned Partha Kamal [Atlassian] Major Open Unresolved Dec 27, 2007 Jul 02, 2009
Bug CONF-10868 Node that can not join cluster due to license restriction causes cluster panic Unassigned Ivan Benko [Atlassian] Major Open Unresolved Feb 29, 2008 Sep 03, 2008
Task CONF-12345 Park issue :) Unassigned Dave Loeng [Atlassian] Major Open Unresolved Jul 07, 2008 Jan 21, 2009
Bug CONF-12486 ClassNotFoundException logged on cluster node startup Unassigned Anatoli Kazatchkov [Atlassian] Major Open Unresolved Jul 17, 2008 Aug 25, 2009
Bug CONF-12614 Intermittent ConcurrentModificationException in cluster Unassigned Anatoli Kazatchkov [Atlassian] Major Open Unresolved Jul 29, 2008 Mar 31, 2009
Bug CONF-13421 Layout customisations are not propagated to other cluster nodes Unassigned Matt Ryall [Atlassian] Major Open Unresolved Oct 16, 2008 Dec 09, 2008
Bug CONF-14657 Retrieving the global settings in a clustered environment causes a lot of contention Unassigned Chris Kiehl [Atlassian] Major Open Unresolved Feb 21, 2009 Nov 08, 2009
Task CONF-15523 Run cluster performance build on two machines Unassigned Matt Ryall [Atlassian] Major Open Unresolved May 05, 2009 May 12, 2010
Bug CONF-16419 Installing a font for PDF export in a cluster will not carry to cluster nodes that are down or unavailable. Unassigned Charles Miller [Atlassian] Major Open Unresolved Jul 20, 2009 Aug 05, 2009
Bug CONF-17089 Reindexing in cluster only runs on one node if triggered from web UI Unassigned Anatoli Kazatchkov [Atlassian] Major Open Unresolved Oct 01, 2009 May 12, 2010
Bug CONF-22548 IllegalArgumentException Encountered when Comparing Page Versions in Cluster Unassigned Tim Wong [Atlassian] Major Open Unresolved May 19, 2011 May 22, 2011
Bug CONF-23033 Viewfile macro does not work in Confluence Clustered when Office Connector is configured to use Cache in Memory for temporary storage Unassigned Roy Hartono [Atlassian] Major Open Unresolved Aug 09, 2011 Aug 09, 2011
Bug CONF-17040 Cannot build milestones outside Atlassian due to coherence Unassigned Jonathan Gilbert [Atlassian] Minor Open Unresolved Sep 25, 2009 Dec 08, 2009
Bug CONF-7368 Confluence Cluster setup dies horribly when DNS fails Unassigned Don Willis [Atlassian] Minor Open Unresolved Nov 23, 2006 Jan 19, 2009
Bug CONF-9020 Cluster nodes do not get notified of Layout changes Unassigned Roberto Dominguez Minor Open Unresolved Jul 27, 2007 Oct 08, 2007
Bug CONF-9059 Cluster setup network interface selection shows loopback interface Unassigned Matt Ryall [Atlassian] Minor Open Unresolved Aug 01, 2007 Aug 01, 2007
Bug CONF-9281 Plugin's I18n properties not loaded in other cluster nodes unless restarted Unassigned Roberto Dominguez Minor Open Unresolved Aug 26, 2007 Apr 16, 2011
Third-party issue CONF-9749 Coherence does not allow the disabling of all JDK shutdown hooks Unassigned Christopher Owen [Atlassian] Minor Open Unresolved Oct 17, 2007 Jan 29, 2008
Bug CONF-13698 Changing custom html on one node of a cluster is not imideatly reflected on the other node. Unassigned Anatoli Kazatchkov [Atlassian] Minor Open Unresolved Nov 12, 2008 Nov 13, 2008
Bug CONF-13870 After a site Import into a cluster, admin console displays attachment storage as filesystem Unassigned Agnes Ro [Atlassian] Minor Open Unresolved Nov 27, 2008 Sep 04, 2011
Bug CONF-17577 Cluster build passed but didn't close down Confluence Unassigned Brian Nguyen [Atlassian] Minor Open Unresolved Nov 10, 2009 May 12, 2010
Bug CONF-22466 Content Permission changes are propagated between nodes one at a time, should be in bulk Unassigned Richard Atkins [Atlassian] Minor Open Unresolved May 09, 2011 May 10, 2011
Bug CONF-22979 Migrating to a cluster with existing data does not add cluster attributes to the confluence.cfg.xml Unassigned Adam Laskowski [Atlassian] Minor Open Unresolved Jul 27, 2011 Jul 28, 2011
Bug CONF-9846 Tangosol configuration: the (optional) cluster-name element is in the wrong place Unassigned Don Willis [Atlassian] Trivial Open Unresolved Oct 30, 2007 Dec 20, 2007
Bug CONF-14088 Locking on cache keys needs to check if the lock was actually aquired Unassigned Chris Kiehl [Atlassian] Trivial Open Unresolved Dec 30, 2008 Nov 22, 2009
Bug CONF-8300 Cannot override TTL configuration through tangosol coherence properties Unassigned Matthew Jensen [Atlassian] Major Needs Verification Unresolved Apr 20, 2007 Nov 10, 2009

Contact Atlassian support

We have dedicated staff on hand to support your installation of Confluence. Please follow the instructions for raising a support request and mention that you're having trouble setting up your Confluence cluster.

Cluster Safety Mechanism


list-interfaces.jar (application/x-java)
Document generated by Confluence on Sep 19, 2011 02:40