NAME

rwsilk2ipfix - Convert SiLK Flow records to IPFIX records

SYNOPSIS

rwsilk2ipfix [--ipfix-output=PATH] [--no-site-name-elements]
      [--print-statistics] [--single-template]
      [--site-config-file=FILENAME]
      {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

rwsilk2ipfix --help

rwsilk2ipfix --version

DESCRIPTION

rwsilk2ipfix reads SiLK Flow records, converts the records to an IPFIX (Internet Protocol Flow Information eXport) format, and writes the IPFIX records to the path specified by --ipfix-output or to the standard output when the --ipfix-output switch is not provided and standard output is not the terminal.

rwsilk2ipfix reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use - or stdin as a file name. If an input file name ends in .gz, the file is uncompressed as it is read. When the --xargs switch is provided, rwsilk2ipfix reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

The IPFIX records generated by rwsilk2ipfix will contain ten information elements that are in the Private Enterprise space for CERT (the IPFIX Private Enterprise Number of CERT is 6871). These ten information elements fall into three groups:

For each of the ten information elements that rwsilk2ipfix produces, the following table lists its numeric ID, its length in octets (or v for variable length), its name, the field name it corresponds to on rwcut(1), and a brief description.

 30  1  silkFlowtypeId    class & type   How rwflowpack categorized
                                         the flow record
 31  2  silkSensorId      sensor         Sensor where the flow was
                                         collected
938  v  silkFlowtypeName  -              Name of the silkFlowtypeId
                                         as read from F<silk.conf>
939  v  silkClassName     class          Class name derived from
                                         the silkFlowtypeId
940  v  silkTypeName      type           Type name derived from the
                                         silkFlowtypeId
941  v  silkSensorName    sensor         Name of the silkSensorId
                                         as read from F<silk.conf>
 14  1  initialTCPFlags   initialFlags   TCP flags on first packet in
                                         the flow record
 15  1  unionTCPFlags     sessionFlags   TCP flags on all packets in
                                         the flow except the first
 32  1  silkTCPState      attributes     Flow continuation attributes
                                         set by generator
 33  2  silkAppLabel      application    Guess by flow generator as
                                         to the content of traffic

Note: Elements 30 and 31, silkFlowtypeId and silkSensorId, may appear as silkFlowType and silkFlowSensor in some documentation.

Templates

Since SiLK 3.12.0, rwsilk2ipfix has used ten different IPFIX templates for writing SiLK Flow records. The --single-template switch causes rwsilk2ipfix to revert to its previous behavior and use a single template for all records.

In SiLK 3.20.0, four additional elements (Elements 938--941) providing the names of the SiLK class, type, and sensor were added to templates used for multi-template output. These elements are variable length and they are not included if the site configuration file (silk.conf(5)) is not available or if the --no-site-name-elements option is given.

SiLK 3.23.0 changed the elements used for exporting the timestamps to flowStartMicroseconds and flowEndMicroseconds. Previously the templates used millisecond timestamps. This change does not apply to the --single-template output.

  1. Template ID 0x9DD0 (40400), for IPv4 records whose protocol is not ICMP, ICMPv6, UDP, SCTP, or TCP:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 38    sourceIPv4Address (8)             12    sIP
     39- 42    destinationIPv4Address (12)       13    dIP
     43- 46    ipNextHopIPv4Address (15)         14    nhIP
     47-  v    silkFlowtypeName (6871, 938)      15    class & type
      v-  v    silkClassName (6871, 939)         16    class
      v-  v    silkTypeName (6871, 940)          17    type
      v-  v    silkSensorName (6871, 941)        18    sensor
  2. Template ID 0x9DD1 (40401), for ICMP IPv4 records:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 36    icmpTypeCodeIPv4                  12    dPort
     37- 40    sourceIPv4Address (8)             13    sIP
     41- 44    destinationIPv4Address (12)       14    dIP
     45- 48    ipNextHopIPv4Address (15)         15    nhIP
     49-  v    silkFlowtypeName (6871, 938)      16    class & type
      v-  v    silkClassName (6871, 939)         17    class
      v-  v    silkTypeName (6871, 940)          18    type
      v-  v    silkSensorName (6871, 941)        19    sensor
  3. Template ID 0x9DD2 (40402), for IPv4 records whose protocol is UDP or SCTP:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 36    sourceTransportPort (7)           12    sPort
     37- 38    destinationTransportPort (11)     13    dPort
     39- 42    sourceIPv4Address (8)             14    sIP
     43- 46    destinationIPv4Address (12)       15    sIP
     47- 50    ipNextHopIPv4Address (15)         16    nhIP
     51-  v    silkFlowtypeName (6871, 938)      17    class & type
      v-  v    silkClassName (6871, 939)         18    class
      v-  v    silkTypeName (6871, 940)          19    type
      v-  v    silkSensorName (6871, 941)        20    sensor
  4. Template ID 0x9DD3 (40403), for IPv4 records whose protocol is TCP and that do not have the expanded TCP flags fields (initial flags and session flags):

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35        tcpControlBits (6)                12    flags
     36- 37    sourceTransportPort (7)           13    sPort
     38- 39    destinationTransportPort (11)     14    dPort
     40- 43    sourceIPv4Address (8)             15    sIP
     44- 47    destinationIPv4Address (12)       16    dIP
     48- 51    ipNextHopIPv4Address (15)         17    nhIP
     52-  v    silkFlowtypeName (6871, 938)      18    class & type
      v-  v    silkClassName (6871, 939)         19    class
      v-  v    silkTypeName (6871, 940)          20    type
      v-  v    silkSensorName (6871, 941)        21    sensor
  5. Template ID 0x9DD4 (40404), for IPv4 records whose protocol is TCP and that have have the initial flags and session flags fields:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35        initialTCPFlags (6871, 14)        12    initialFlags
     36        unionTCPFlags (6871, 15)          13    sessionFlags
     37        tcpControlBits (6)                14    flags
     38- 39    sourceTransportPort (7)           15    sPort
     40- 41    destinationTransportPort (11)     16    dPort
     42- 45    sourceIPv4Address (8)             17    sIP
     46- 49    destinationIPv4Address (12)       18    dIP
     50- 53    ipNextHopIPv4Address (15)         19    nhIP
     54-  v    silkFlowtypeName (6871, 938)      20    class & type
      v-  v    silkClassName (6871, 939)         21    class
      v-  v    silkTypeName (6871, 940)          22    type
      v-  v    silkSensorName (6871, 941)        23    sensor
  6. Template ID 0x9ED0 (40656), for IPv6 records whose protocol is not ICMP, ICMPv6, UDP, SCTP, or TCP:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 50    sourceIPv6Address (27)            12    sIP
     51- 66    destinationIPv6Address (28)       13    dIP
     67- 82    ipNextHopIPv6Address (62)         14    nhIP
     83-  v    silkFlowtypeName (6871, 938)      15    class & type
      v-  v    silkClassName (6871, 939)         16    class
      v-  v    silkTypeName (6871, 940)          17    type
      v-  v    silkSensorName (6871, 941)        18    sensor
  7. Template ID 0x9ED1 (40657), for ICMPv6 IPv6 records:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 36    icmpTypeCodeIPv6                  12    dPort
     37- 52    sourceIPv6Address (27)            13    sIP
     53- 68    destinationIPv6Address (28)       14    dIP
     69- 84    ipNextHopIPv6Address (62)         15    nhIP
     85-  v    silkFlowtypeName (6871, 938)      16    class & type
      v-  v    silkClassName (6871, 939)         17    class
      v-  v    silkTypeName (6871, 940)          18    type
      v-  v    silkSensorName (6871, 941)        19    sensor
  8. Template ID 0x9ED2 (40658), for IPv6 records whose protocol is UDP or SCTP:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35- 36    sourceTransportPort (7)           12    sPort
     37- 38    destinationTransportPort (11)     13    dPort
     39- 54    sourceIPv6Address (27)            14    sIP
     55- 70    destinationIPv6Address (28)       15    dIP
     71- 86    ipNextHopIPv6Address (62)         16    nhIP
     87-  v    silkFlowtypeName (6871, 938)      17    class & type
      v-  v    silkClassName (6871, 939)         18    class
      v-  v    silkTypeName (6871, 940)          19    type
      v-  v    silkSensorName (6871, 941)        20    sensor
  9. Template ID 0x9ED3 (40659), for IPv6 records whose protocol is TCP and that do not have the expanded TCP flags fields (initial flags and session flags):

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35        tcpControlBits (6)                12    flags
     36- 37    sourceTransportPort (7)           13    sPort
     38- 39    destinationTransportPort (11)     14    dPort
     40- 55    sourceIPv6Address (27)            15    sIP
     56- 71    destinationIPv6Address (28)       16    dIP
     72- 87    ipNextHopIPv6Address (62)         17    nhIP
     88-  v    silkFlowtypeName (6871, 938)      18    class & type
      v-  v    silkClassName (6871, 939)         19    class
      v-  v    silkTypeName (6871, 940)          20    type
      v-  v    silkSensorName (6871, 941)        21    sensor
  10. Template ID 0x9ED4 (40660), for IPv6 records whose protocol is TCP and that have have the initial flags and session flags fields:

     OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
    =======    =============================    ===    =============
      0-  7    flowStartMicroseconds (152)        1    sTime
      8- 15    flowEndMicroseconds (153)          2    eTime
     16- 19    packetDeltaCount (2)               3    packets
     20- 23    octetDeltaCount (1)                4    bytes
     24- 25    ingressInterface (10)              5    in
     26- 27    egressInterface (14)               6    out
     28- 29    silkAppLabel (6871, 33)            7    application
     30- 31    silkSensorId (6871, 31)            8    sensor
     32        silkFlowtypeId (6871, 30)          9    class & type
     33        silkTCPState (6871, 32)           10    attributes
     34        protocolIdentifier (4)            11    protocol
     35        initialTCPFlags (6871, 14)        12    initialFlags
     36        unionTCPFlags (6871, 15)          13    sessionFlags
     37        tcpControlBits (6)                14    flags
     38- 39    sourceTransportPort (7)           15    sPort
     40- 41    destinationTransportPort (11)     16    dPort
     42- 57    sourceIPv6Address (27)            17    sIP
     58- 73    destinationIPv6Address (28)       18    dIP
     74- 89    ipNextHopIPv6Address (62)         19    nhIP
     90-  v    silkFlowtypeName (6871, 938)      20    class & type
      v-  v    silkClassName (6871, 939)         21    class
      v-  v    silkTypeName (6871, 940)          22    type
      v-  v    silkSensorName (6871, 941)        23    sensor

When the --single-template switch is provided, rwipfix2silk uses a single IPFIX template for all records. That template has ID 0xAFEA (45034) and contains the following information elements:

 OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
=======    =============================    ===    =============
  0-  7    flowStartMilliseconds (152)        1    sTime
  8- 15    flowEndMilliseconds (153)          2    eTime
 16- 31    sourceIPv6Address (27)             3    sIP
 32- 47    destinationIPv6Address (28)        4    dIP
 48- 51    sourceIPv4Address (8)              5    sIP
 52- 55    destinationIPv4Address (12)        6    dIP
 56- 57    sourceTransportPort (7)            7    sPort
 58- 59    destinationTransportPort (11)      8    dPort
 60- 63    ipNextHopIPv4Address (15)          9    nhIP
 64- 79    ipNextHopIPv6Address (62)         10    nhIP
 80- 83    ingressInterface (10)             11    in
 84- 87    egressInterface (14)              12    out
 88- 95    packetDeltaCount (2)              13    packets
 96-103    octetDeltaCount (1)               14    bytes
104        protocolIdentifier (4)            15    protocol
105        silkFlowtypeId (6871, 30)         16    class & type
106-107    silkSensorId (6871, 31)           17    sensor
108        tcpControlBits (6)                18    flags
109        initialTCPFlags (6871, 14)        19    initialFlags
110        unionTCPFlags (6871, 15)          20    sessionFlags
111        silkTCPState (6871, 32)           21    attributes
112-113    silkAppLabel (6871, 33)           22    application
114-119    paddingOctets (210)               23    -

Note that the template contains both IPv4 and IPv6 addresses. One set of those addresses contains the IP addresses and the other set contains only zeros. The template never includes elements 938--941.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--ipfix-output=PATH

Write the IPFIX records to PATH, where PATH is a filename, a named pipe, the keyword stderr to write the output to the standard error, or the keyword stdout or - to write the output to the standard output. If PATH names an existing file, rwsilk2ipfix exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwipfix2silk to exit with an error.

--no-site-name-elements

Do not export the elements that use the site configuration file (silk.conf(5)) to get the names of the flowtype, class, type, and sensor. That is, do not include silkFlowtypeName, silkClassName, silkTypeName, and silkSensorName in the exported templates and records. Since SiLK 3.20.0.

Print, to the standard error, the number of records that were written to the IPFIX output file.

--single-template

Use a single IPFIX template for all records. Using this switch produces output identical to that produced by rwsilk2ipfix from SiLK 3.11.0 and earlier. Since SiLK 3.12.0.

--site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwsilk2ipfix searches for the site configuration file in the locations specified in the "FILES" section.

--xargs
--xargs=FILENAME

Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwsilk2ipfix opens each named file in turn and reads records from it as if the filenames had been listed on the command line.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ($) represents the shell prompt. The text after the dollar sign represents the command line.

To convert the SiLK file silk.rw into an IPFIX format and store the results in ipfix.dat:

$ rwsilk2ipfix --ipfix-output=ipfix.dat silk.rw

To view the contents of ipfix.dat using the yafscii(1) tool (see http://tools.netsa.cert.org/yaf/):

$ yafscii --in=ipfix.dat --out=-

To view the contents of ipfix.dat using the ipfixDump(1) tool (see http://tools.netsa.cert.org/yaf/):

$ ipfixDump --yaf --in=ipfix.dat --out=-

Use the rwipfix2silk(1) tool to convert the IPFIX file back into SiLK Flow format:

$ rwipfix2silk --silk-output=silk2.rw ipfix.dat

ENVIRONMENT

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_CONFIG_FILE

This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwsilk2ipfix may use this environment variable when searching for the SiLK site configuration file.

SILK_PATH

This environment variable gives the root of the install tree. When searching for configuration files, rwsilk2ipfix may use this environment variable. See the "FILES" section for details.

FILES

${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/share/silk/silk.conf
/usr/share/silk.conf

Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.

SEE ALSO

rwipfix2silk(1), rwcut(1), rwflowpack(8), silk.conf(5), silk(7), yaf(1), yafscii(1), ipfixDump(1), applabel(1)