Home > Uncategorized > Load Balancing with Rsyslog

Load Balancing with Rsyslog

I’ve been playing with rsyslog, which I want to use as a cheap (free!), self-contained means of load balancing logs across multiple log receivers, from multiple log sources.

RSyslog is very powerful (some would call it complex 🙂 ), but I got there. As always, my use case was not a typical one for the protocol, so few examples of what I wanted to do, but I found a way using timestamps.

Rather than using uptime, which has 1-second resolution, I’m used timegenerated, which has microsecond resolution. I was concerned that running at high rates and large numbers (eg. 100keps across 100 devices) would result in each device being blasted by 100keps for 1 second.

This way, I flip every microsecond to a new destination. I chose not to use the event time, since it may not have subsecond resolution.  I also chose not to use the event index number that I get inside every event (I have one), since it would require regex of the form .*?EventId=(\d+).*?, which could be expensive in large (6-8KB) log lines, and slow down processing.

Next step is to go from UDP to Encrypted TCP, which should hopefully be easy.

Code snippet for the load balancing (across 3 in test);

# Load balance output based on system time
# Define a template that contains just the subsecond value of the event receipt time (not event timestamp!).
# For timegenerated, this is in microseconds
template(name="subseconds" type="string" string="%timegenerated:::date-subseconds%")
# Set a variable to that value
set $!subsecs = exec_template("subseconds");
# Perform a modulo of the subsecond value of the receipt time to decide which way to send it
if ($!subsecs % 3 == 0) then call output_0
if ($!subsecs % 3 == 1) then call output_1
if ($!subsecs % 3 == 2) then call output_2

Categories: Uncategorized
  1. June 10, 2015 at 6:48 pm

    just curious, how did this turn out? I am looking at udp load balancing and this seems like a simpler solution.

    • damo
      June 12, 2015 at 3:20 am

      It works – I was able to load balance across several Syslog receivers. I haven’t tried volume yet, but I’ve had others say they achieved 30,000 messages per second easily.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: