Monitoring for Large-Scale Networks

SiLK

int-ext-fields

NAME

int-ext-fields - SiLK plug-in providing internal/external ip/port fields

SYNOPSIS

rwcut --plugin=int-ext-fields.so --fields=FIELDS ...

rwgroup --plugin=int-ext-fields.so --fields=FIELDS ...

rwsort --plugin=int-ext-fields.so --fields=FIELDS ...

rwstats --plugin=int-ext-fields.so --fields=FIELDS ...

rwuniq --plugin=int-ext-fields.so --fields=FIELDS ...

DESCRIPTION

The int-ext-fields plug-in adds four potential fields to rwcut(1), rwgroup(1), rwsort(1), rwstats(1), and rwuniq(1). These fields contain the internal IP (int-ip), the external IP (ext-ip), the internal port (int-port, and the external port (ext-port). To use these fields, specify their names in the --fields switch.

These fields can be useful when a file contains flow records that were collected for multiple directions---for example, some flow records are incoming and some are outgoing.

For these fields to be available, the user must specify the list of flowtypes (i.e., class/type pairs) that are considered incoming and the list that are considered outgoing. The user must specify the flowtypes because SiLK has no innate sense of the direction of a flow record. Although "in" and "out" are common types, SiLK does not recognize that these represent flows going in opposite directions.

If a record has a flowtype that is not in the list of incoming and output flowtypes, the application uses a value of 0 for that field.

The user specifies the flowtypes by giving a comma-separated list of class/type pairs using the --incoming-flowtypes and --outgoing-flowtypes switches on the application's command line. When the switch is not provided, the application checks the INCOMING_FLOWTYPES and OUTGOING_FLOWTYPES environment variables. If the list of incoming and/or outgoing flowtypes are not specified, the fields are not available.

For the packlogic-twoway(3) site, one would set the following environment variables:

INCOMING_FLOWTYPES=all/in,all/inweb,all/inicmp,all/innull
OUTGOING_FLOWTYPES=all/out,all/outweb,all/outicmp,all/outnull

The parsing of flowtypes requires the silk.conf(5) site configuration file. You may need to set the SILK_CONFIG_FILE environment variable or specify --site-config-file on the command line prior to loading the plug-in.

OPTIONS

The int-ext-fields plug-in provides the following options to rwcut, rwgroup, rwsort, rwstats, and rwuniq.

--fields=FIELDS

FIELDS refers to a list of fields to use for the operation. The int-ext-fields plug-in adds the following fields for display, sorting, and grouping using the rwcut(1), rwgroup(1), rwsort(1), rwstats(1), and rwuniq(1) tools:

int-ip: Print, sort by, or group by the internal IP address. The internal IP is the destination address for incoming flowtypes and the source address for outgoing flowtypes. When a SiLK Flow record's flowtype is not listed in either the incoming or outgoing flowtypes list, the int-ip field is 0.
ext-ip: Print, sort by, or group by the external IP address. The external IP is the source address for incoming flowtypes and the destination address for outgoing flowtypes. When a SiLK Flow record's flowtype is not listed in either the incoming or outgoing flowtypes list, the ext-ip field is 0.
int-port: Print, sort by, or group by the internal port. This value is 0 for ICMP flow records, and when the SiLK Flow record's flowtype is not listed in either the incoming or outgoing flowtypes list.
ext-port: Print, sort by, or group by the external port. This value is 0 for ICMP flow records, and when the SiLK Flow record's flowtype is not listed in either the incoming or outgoing flowtypes list.

--incoming-flowtypes=CLASS/TYPE[,CLASS/TYPE ...]

Names the flowtypes that should be considered incoming. The list of flowtypes should be specified as a comma-separated list of class/type pairs. This switch overrides the flowtype list specified in the INCOMING_FLOWTYPES environment variable. If this switch is not provided and the INCOMING_FLOWTYPES environment variable is not set, the int-ext-fields plug-in will not define any fields.

--outgoing-flowtypes=CLASS/TYPE[,CLASS/TYPE ...]

Similar to --incoming-flowtypes, except it names the flowtypes that should be considered outgoing, and it overrides the OUTGOING_FLOWTYPES environment variable.

EXAMPLE

In the following example, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash (\) is used to indicate a wrapped line.

Consider the file data.rw that contains data going in different directions:

$ rwcut --fields=sip,sport,dip,dport,proto,class,type data.rw
            sIP|sPort|            dIP|dPort|pro|cla|   type|
   10.239.86.13|29897|192.168.228.153|   25|  6|all|     in|
192.168.228.153|   25|   10.239.86.13|29897|  6|all|    out|
192.168.208.237|29416| 10.233.108.250|   25|  6|all|    out|
 10.233.108.250|   25|192.168.208.237|29416|  6|all|     in|
 192.168.255.94|29301|  10.198.18.193|   80|  6|all| outweb|
  10.198.18.193|   80| 192.168.255.94|29301|  6|all|  inweb|
   10.202.7.122|29438|192.168.248.202|   25|  6|all|     in|
192.168.248.202|   25|   10.202.7.122|29438|  6|all|    out|
 10.255.142.104|26731|192.168.236.220|   25|  6|all|     in|
192.168.236.220|   25| 10.255.142.104|26731|  6|all|    out|

Using the int-ext-fields plug-in allows one to print the internal and external addresses and ports (note: command line wrapped for improved readability):

$ rwcut --plugin=int-ext-fields.so                                 \
       --incoming=all/in,all/inweb --outgoing=all/out,all/outweb   \
       --fields=ext-ip,ext-port,int-ip,int-port,proto,class,type
         ext-ip|ext-p|         int-ip|int-p|pro|cla|   type|
   10.239.86.13|29897|192.168.228.153|   25|  6|all|     in|
   10.239.86.13|29897|192.168.228.153|   25|  6|all|    out|
 10.233.108.250|   25|192.168.208.237|29416|  6|all|    out|
 10.233.108.250|   25|192.168.208.237|29416|  6|all|     in|
  10.198.18.193|   80| 192.168.255.94|29301|  6|all| outweb|
  10.198.18.193|   80| 192.168.255.94|29301|  6|all|  inweb|
   10.202.7.122|29438|192.168.248.202|   25|  6|all|     in|
   10.202.7.122|29438|192.168.248.202|   25|  6|all|    out|
 10.255.142.104|26731|192.168.236.220|   25|  6|all|     in|
 10.255.142.104|26731|192.168.236.220|   25|  6|all|    out|

This can be especially useful when using a tool like rwuniq or rwstats:

$ export INCOMING_FLOWTYPES=all/in,all/inweb
$ export OUTGOING_FLOWTYPES=all/out,all/outweb
$ rwuniq --plugin=int-ext-fields.so               \
       --fields=int-ip,int-port --value=bytes
         int-ip|int-p|               Bytes|
192.168.208.237|29416|               28517|
192.168.248.202|   25|                4016|
192.168.228.153|   25|                3454|
192.168.236.220|   25|               31872|
 192.168.255.94|29301|               14147|

Beware of traffic whose type is not listed in INCOMING_FLOWTYPES or OUTGOING_FLOWTYPES

$ rwcut --num-rec=4 --fields=sip,sport,dip,dport,proto,type data2.rw
            sIP|sPort|            dIP|dPort|pro|   type|
     67.215.0.5|   53|      155.6.5.1| 1613| 17|ext2ext|
     67.215.0.5|   53|      155.6.5.1| 1895| 17|ext2ext|
     67.215.0.5|   53|      155.6.5.1| 1351| 17|ext2ext|
     67.215.0.5|   53|      155.6.5.1| 1988| 17|ext2ext|

since the int-ext-fields plug-in sets the fields to 0.

$ rwcut --num-rec=4 --plugin=int-ext-fields.so                     \
       --incoming=all/in,all/inweb --outgoing=all/out,all/outweb   \
       --fields=int-ip,int-port,ext-ip,ext-port,proto,type data4.rw
         int-ip|int-p|         ext-ip|ext-p|pro|   type|
        0.0.0.0|    0|        0.0.0.0|    0| 17|ext2ext|
        0.0.0.0|    0|        0.0.0.0|    0| 17|ext2ext|
        0.0.0.0|    0|        0.0.0.0|    0| 17|ext2ext|
        0.0.0.0|    0|        0.0.0.0|    0| 17|ext2ext|

ENVIRONMENT

INCOMING_FLOWTYPES

Used as the value for the --incoming-flowtypes when that switch is not provided.

OUTGOING_FLOWTYPES

Used as the value for the --outgoing-flowtypes when that switch is not provided.

SILK_CONFIG_FILE

This environment variable is used when the SiLK application attempts to locate the the SiLK site configuration file unless the --site-config-file switch is specified. Additional locations where the application searches are listed in the "FILES" section. The site configuration file is required to parse the flowtypes.

SILK_DATA_ROOTDIR

This environment variable specifies the root directory of data repository. As described in the "FILES" section, an application may use this environment variable when searching for the SiLK site configuration file.

SILK_PATH

This environment variable gives the root of the install tree. When searching for configuration files and plug-ins, an application may use this environment variable. See the "FILES" section for details.

SILK_PLUGIN_DEBUG

When set to 1, the SiLK applications print status messages to the standard error as they attempt to find and open the int-ext-fields.so plug-in. A typical invocation using this variable is

env SILK_PLUGIN_DEBUG=1 rwcut --plugin=int-ext-fields.so --version

FILES

${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf: Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.
${SILK_PATH}/lib64/silk/int-ext-fields.so
${SILK_PATH}/lib64/int-ext-fields.so
${SILK_PATH}/lib/silk/int-ext-fields.so
${SILK_PATH}/lib/int-ext-fields.so
/usr/lib64/silk/int-ext-fields.so
/usr/lib64/int-ext-fields.so
/usr/lib/silk/int-ext-fields.so
/usr/lib/int-ext-fields.so: Possible locations for the plug-in.

BUGS

The int-ip and ext-ip fields do not respect the --ip-format switch nor SILK_IP_FORMAT environment variable. The IP addresses are printed in the canonical format, and the columns are wide enough for an IPv6 address.