Load Balancing with Rsyslog
I’ve been playing with rsyslog, which I want to use as a cheap (free!), self-contained means of load balancing logs across multiple log receivers, from multiple log sources.
RSyslog is very powerful (some would call it complex 🙂 ), but I got there. As always, my use case was not a typical one for the protocol, so few examples of what I wanted to do, but I found a way using timestamps.
Rather than using uptime, which has 1-second resolution, I’m used timegenerated, which has microsecond resolution. I was concerned that running at high rates and large numbers (eg. 100keps across 100 devices) would result in each device being blasted by 100keps for 1 second.
This way, I flip every microsecond to a new destination. I chose not to use the event time, since it may not have subsecond resolution. I also chose not to use the event index number that I get inside every event (I have one), since it would require regex of the form .*?EventId=(\d+).*?, which could be expensive in large (6-8KB) log lines, and slow down processing.
Next step is to go from UDP to Encrypted TCP, which should hopefully be easy.
Code snippet for the load balancing (across 3 in test);
# Load balance output based on system time
# Define a template that contains just the subsecond value of the event receipt time (not event timestamp!).
# For timegenerated, this is in microseconds
template(name="subseconds" type="string" string="%timegenerated:::date-subseconds%")
# Set a variable to that value
set $!subsecs = exec_template("subseconds");
# Perform a modulo of the subsecond value of the receipt time to decide which way to send it
if ($!subsecs % 3 == 0) then call output_0
if ($!subsecs % 3 == 1) then call output_1
if ($!subsecs % 3 == 2) then call output_2