How can redundancy be established for producer nodes?

Question

It is rather straightforward to create relay redundancy but it is not as clear how to create redundancies for the producer nodes. The best solution I could come up with for the time being are VMware ESXi server farms, over SAN connections, with CARP IPs, etc. creating a physical redundancy but the OS layer remains a single point of failure.

I believe that running redundant producer nodes is not something approved.

Is there anyway to create an Active/Active or Active/Passive environment for the producer nodes?

Answer 1

Currently, a block producer can only send out its blocks if there is an incoming connection to it. So, as long as your backup block producer has no incoming connections, it can be "hot" and ready to produce blocks. I recommend having a relay pointing to a backup block producer, but prevented from connecting by a firewall rule.

Use CNCLI ping to determine if the main block producer is up or down. Depending on the answer, you can add/remove the firewall block to the backup block producer.

Example script to run with cron once per minute. In this example, a backup relay and block producer are installed on the same server.

#!/bin/bash

# are we blocking? In other words, we are not in failover mode
isblocking=`/usr/sbin/iptables --list INPUT --numeric | grep REJECT | grep 5050 | wc -l`

#if [[ $isblocking -eq 1 ]]
#then
#  echo "NON-failover mode. checking..."
#else
#  echo "FAILOVER mode. checking..."
#fi

error=`/home/westbam/.cargo/bin/cncli ping --host relay0.myserver.com --port 5000 | jq .status | grep error | wc -l`
if [[ $error -eq 1 ]]
then
  #echo "relay0 error. check relay1..."
  sleep 10
  error=`/home/westbam/.cargo/bin/cncli ping --host relay1.myserver.com --port 5000 | jq .status | grep error | wc -l`
  if [[ $error -eq 1 ]]
  then
    #echo "relay1 error. check relay2..."
    sleep 10
    error=`/home/westbam/.cargo/bin/cncli ping --host relay2.myserver.com --port 5000 | jq .status | grep error | wc -l`
    if [[ $error -eq 1 ]]
    then
      if [[ $isblocking -eq 1 ]]
      then
        echo "$(date): Enter FAILOVER mode..."
        # remove the local node blocks
        /usr/sbin/iptables --delete INPUT -p tcp --dport 5050 --jump REJECT 2> /dev/null
        /usr/bin/mail -s "FAILOVER" [email protected] <<< "FAILOVER to relay3 enabled"
      fi
      exit 0
    fi
  fi
fi

if [[ $isblocking -eq 0 ]]
then
  # we're not blocking local traffic, but we SHOULD be. Turn on the blocks
  echo "$(date): Return to NORMAL mode..."
  /usr/sbin/iptables --insert INPUT -p tcp --dport 5050 --jump REJECT 2> /dev/null
  /usr/bin/mail -s "NORMAL" [email protected] <<< "failover disabled"
fi

Answer 2

Any non-block-producer full node a passive node. Just load your keys and you now have your block producer.

You can also use any OS specific high-available software, such as pecemaker or corosync.