Quick Setup Guide for super_mediator

This tutorial is a step-by-step guide of setting up YAF and super_mediator. For a detailed tutorial on super_mediator and the different configuration options, see this tutorial. This particular tutorial takes the DPI data produced by yaf and imports the data into a MariaDB or MySQL database. This also enables DNS deduplication in super_mediator. This tutorial does not include an example of how to setup SiLK; for that, see this page.

Install Procedure

This provides a brief overview of the installation procedure when building from source. For more detailed information see each tool's installation instructions.

Install Prerequisites

yum groupinstall "Development Tools"
yum install libpcap-devel pcre-devel

Install either MariaDB:

yaf install mariadb-server mariadb-devel

or MySQL:

yaf install mysql-server mysql-devel

Build and install libfixbuf:

tar -xvzf libfixbuf-3.0.0.tar.gz
cd libfixbuf-3.0.0
./configure
make
make install

Install YAF and Super Mediator

NOTE: Installing from source will overwrite previous versions of YAF's and Super Mediator's configuration files in the /usr/local/etc directory (the location may be different depending on the options given to configure). If you have customized those files, make a copy prior to installing a new versions of YAF and Super Mediator.

Build and install YAF. The minimum recommended options are shown here; see the YAF installation page for other options.

tar -xvzf yaf-3.0.0.tar.gz
cd yaf-3.0.0
./configure --enable-applabel --dpi
make
make install

Build and install super_mediator:

tar -xvzf super_mediator-2.0.0.tar.gz
cd super_mediator-2.0.0
./configure --with-mysql
make
make install

Setup MariaDB or MySQL

Start the database service:

service mariadb start

or

service mysqld start

Set a password for the root user:

/usr/bin/mysqladmin -u root password '<SuperSecretPassword>'

Login to the database. It will prompt you for the password you created in the previous step:

mysql -u root -p

Create the database you intend to use for super_mediator:

mysql> create database smediator;

Create a user for super_mediator to access the database:

mysql> CREATE USER 'mediator'@'localhost' IDENTIFIED BY '<SuperSecretPassword>';

Give permissions to user to access only the smediator database:

mysql> GRANT ALL ON smediator.* TO mediator@'localhost';

Create MariaDB or MySQL Tables

Use super_table_creator to create all the tables in your database:

/usr/local/bin/super_table_creator --name mediator \
    --pass=<SuperSecretPassword> --database=smediator
/usr/local/bin/super_table_creator --name mediator \
    --pass=<SuperSecretPassword> --database=smediator --dns-dedup

Configure super_mediator

Create output directories:

mkdir -p /data/smediator/dpi
mkdir -p /data/smediator/dns

Create your super_mediator.conf file. One is installed by default into /usr/local/etc. (The location may be different depending on how super_mediator is built.) The following one will get you started (you should add your <SuperSecretPassword>):

COLLECTOR TCP
   PORT 18000
COLLECTOR END

#dedup process
EXPORTER TEXT ROTATING_FILES "dedup"
   PATH "/data/smediator/dns/yaf2dns"
   DELIMITER "|"
   ROTATE_INTERVAL 1200
   DNS_DEDUP_ONLY
   LOCK
   MYSQL_USER "mediator"
   MYSQL_PASSWORD "<SuperSecretPassword>"
   MYSQL_TABLE "dns-dedup"
   MYSQL_DATABASE "smediator"
EXPORTER END

#dpi 2 database
EXPORTER TEXT ROTATING_FILES
   PATH "/data/smediator/dpi"
   ROTATE_INTERVAL 1200
   MULTI_FILES
   DPI_ONLY
   LOCK
   MYSQL_USER "mediator"
   MYSQL_PASSWORD "<SuperSecretPassword>"
   MYSQL_DATABASE "smediator"
EXPORTER END

DNS_DEDUP "dedup"
   MAX_HIT_COUNT 5000
DNS_DEDUP END

LOGLEVEL DEBUG
LOG "/var/log/super_mediator.log"
PIDFILE "/data/super_mediator.pid"

Start Tools

Start super_mediator

super_mediator -c /usr/local/etc/super_mediator.conf --daemonize

Confirm super_mediator is running:

ps -ef | grep super

If super_mediator is not running, check for any errors:

cat /var/log/super_mediator.log

Start yaf:

mkdir /var/log/yaf
export LTDL_LIBRARY_PATH=/usr/local/lib/yaf

Run yaf over PCAP file:

/usr/local/bin/yaf \
    --in <PCAP FILE> \
    --ipfix tcp \
    --out localhost \
    --ipfix-port 18000 \
    --log /var/log/yaf/yaf.log \
    --verbose \
    --silk \
    --dpi --max-payload 2048

OR Run yaf on interface eth0:

/usr/local/bin/yaf \
    --in eth0 --live pcap \
    --ipfix tcp \
    --out localhost \
    --ipfix-port 18000 \
    --log /var/log/yaf/yaf.log \
    --verbose \
    --silk \
    --dpi --max-payload 2048 \

For releases of yaf prior to 3.0.0, replace --dpi with the options --applabel --plugin-name=/usr/local/lib/yaf/dpacketplugin.so