Communication

Cluster nodes need to communicate with each other primarily to send cache invalidation messages. Without this communication, the Axon Ivy Cluster cannot function properly. To facilitate this, the Axon Ivy Engine uses the reliable JGroups library. You can fully customize the communication settings by providing your own [ivyEngine]/configuration/jgroups.xml file.

Discovery

First, a discovery mechanism is used to locate other nodes within the cluster. By default, the Axon Ivy Engine uses UDP multicast over port 45588 to discover other nodes. The discovery protocol can be changed if UDP multicast is not suitable for your environment. For example:

  • TCP/IP-based discovery with TCPPING

  • DNS-based discovery, which is often suitable for cloud environments (DNS_PING)

Messaging

After discovery, cluster nodes communicate with each other by sending messages—by default using UDP multicast on port 45588. This is typically the most efficient way to send messages in a local network. However, if your environment does not support UDP multicast, you can change the protocol.

TCP/IP

If UDP multicast is not suitable for your environment, you can switch to TCP/IP. This is often necessary in cloud environments where multicast is not supported. To configure TCP/IP, you can modify the jgroups.xml file to use the TCP transport instead of multicast. Here is an example configuration using DNS-based discovery with TCP/IP as protocol:

jgroups.xml example of a TCP/IP based communication and DNS-based discovery mechanism
 1<config xmlns="urn:org:jgroups"
 2    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 3    xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"
 4>
 5    <!-- Instead of using UDP, we will use TCP/IP as the communication protocol -->
 6    <!-- In terms of performance, UDP is generally more efficient due to its lower overhead compared to TCP -->
 7    <!-- Read more about it here http://jgroups.org/manual5/#_tcp-->
 8    <TCP bind_addr="${jgroups.bind_addr:site_local}"
 9         bind_port="${jgroups.bind_port:7800}"
10         thread_pool.min_threads="0"
11         thread_pool.max_threads="200"
12         thread_pool.keep_alive_time="30000"/>
13    <RED />
14
15    <!-- DNS_PING uses DNS A entries to perform discovery -->
16    <!-- It is also possible to use DNS SRV entries, read more about it here http://jgroups.org/manual5/#_dns_ping -->
17    <!-- Change dns_query according to your environment -->
18    <dns.DNS_PING
19        dns_query="ivy.sandbox.svc.cluster.local" />
20
21    <MERGE3 max_interval="30s"
22        min_interval="10s" />
23    <FD_SOCK2 />
24    <FD_ALL3 />
25    <VERIFY_SUSPECT2 timeout="1.5s" />
26    <BARRIER />
27    <pbcast.NAKACK2 xmit_interval="0.5s" />
28    <UNICAST3 xmit_interval="0.5s" />
29    <pbcast.STABLE desired_avg_gossip="50s"
30        max_bytes="4M" />
31    <pbcast.GMS print_local_addr="true" join_timeout="1s" />
32    <MFC max_credits="4M"
33        min_threshold="0.4" />
34    <UFC max_credits="4M"
35        min_threshold="0.4" />
36    <FRAG2 frag_size="60K" />
37    <pbcast.STATE_TRANSFER />
38</config>

Troubleshooting

If cluster nodes cannot discover each other or you experience messaging issues, you can enable specific Loggers to gain more insight into what is happening.

log4j2.xml configuration for troubleshooting cluster communication
 1<Configuration>
 2    <Appenders>
 3        <Console name="ConsoleLog" target="SYSTEM_OUT">
 4            <PatternLayout pattern="${pattern}" />
 5            <ThresholdFilter level="debug" />
 6        </Console>
 7    </Appenders>
 8    <Loggers>
 9        <Logger name="org.jgroups" level="debug" />
10        <Logger name="ch.ivyteam.ivy.cluster" level="debug" />
11    </Loggers>
12</Configuration>