Notes

Table of examples

Various filter examples (return to top)

The specificDport filter looks for destination port of 11111

FILTER specificDport
    DPORT == 11111
END FILTER

The ftpFilter looks for incoming FTP traffic (TCP port 21)

FILTER ftpFilter
    DPORT ==21
    PROTOCOL ==6
END FILTER

The HighPorts filter looks for TCP traffic with both ports greater than 50000.

FILTER HighPorts
    SPORT >= 50000
    DPORT >= 50000
    PROTOCOL == 6
END FILTER

The writeDownHighPortSips internal filter checks to see if the flow meets the requirements of HighPorts, and puts the SIP into a list called highPortSIPs. Entries in the list are kept for 1 day.

INTERNAL FILTER writeDownHighPorts
    FILTER HighPorts
    SIP highPortSIPs 1 DAY
END INTERNAL FILTER

The sipInHighPortSIPs filter looks for SIPs identified by the above internal filter writeDownHighPorts by checking if they are in the list highPortsSIPs

FILTER sipInHighPortsSIPs
    SIP IN LIST highPortsSIPs
END FILTER

Alert on all flows (return to top)

The simplest configuration file sends an alert with every flow that is seen by Pipeline. The filter allows all flows through, and the evaluation is EVERYTHING PASSES with no alert regulation.

FILTER all
    #No comparisons needed for a filter that lets everything through
END FILTER

This evaluation uses ALERT ALWAYS to send alerts anytime there is something available. ALERT EVERYTHING includes all outputs anytime pipeline is allowed to alert.

Watchlist (return to top)

A standard use of the Analysis Pipeline is to perform watchlisting on a network. For Pipeline, watchlists are stored as IPSets.

Watchlisting means looking for traffic to and/or from a set of known bad IP addresses. There are two ways to look for watchlist traffic:

Use one filter to see if either SIP or DIP is in the watchlist.

FILTER eitherIPInWatchlist
    ANY IP IN LIST "myWatchlist.set"
END FILTER

EVALUATION watchlistEitherDirection
    FILTER eitherIPInWatchlist
    SEVERITY 1
    ALERT ALWAYS
    ALERT EVERYTHING
    CHECK EVERYTHING PASSES
    END CHECK
END EVALUATION

A filter and connected evaluation to look for traffic in each direction

FILTER SIPInWatchlist
    SIP IN LIST "myWatchlist.set"
END FILTER

EVALUATION watchlistSIP
    FILTER SIPInWatchlist
    SEVERITY 1
    ALERT ALWAYS
    ALERT EVERYTHING
    CHECK EVERYTHING PASSES
    END CHECK
END EVALUATION

FILTER DIPInWatchlist
    DIP IN LIST "myWatchlist.set"
END FILTER

EVALUATION watchlistDIP
    FILTER DIPInWatchlist
    SEVERITY 1
    ALERT ALWAYS
    ALERT EVERYTHING
    CHECK EVERYTHING PASSES
    END CHECK
END EVALUATION

The second option allows for more detailed alerts as the name of the evaluation can include the directionality of the watchlist entry

DNS Watchlist (return to top)

A DNS watchlist checks for domain names queried by users in the network that are in a list of known malicious domains. These is different than other watchlists for three reasons. The first is the format of the watchlist which tells Pipeline that the values are to be interpretted as DNS names versus regular strings. The second difference is that the domain names are likely to come from a YAF data source, meaning there could be multiple dnsQNames in each record. The last difference is that if the record is from a YAF data source, the dnsQName is not in the main part of the record, so the evaluation needs to use EXTRA ALERT FIELD to ensure that the dnsQName makes it into the alert.

To tell Pipeline to interpret both the watchlist file's values, and the values from the records as domain names, the first line of the DNS watchlist file must be "##format:dns". Because of this first line, Pipeline will say that "www.cert.org" matches if "cert.org" is in the watchlist. If the field value matches all of the value in the watchlist, even if not all of the value is used, the comparison will return true.

Example file: dnsNames.txt
##format:dns
tools.netsa.cert.org
sei.cmu.edu

FILTER importantDomains
    dnsQName IN LIST "dnsNames.txt"
END FILTER

EVALUATION alertOnDomains
    FILTER importantDomains
    SEVERITY 1
    ALERT ALWAYS
    ALERT EVERYTHING
    CHECK EVERYTHING PASSES
    END CHECK
    EXTRA ALERT FIELD dnsQName
END EVALUATION

Simple Count (return to top)

While it may not be useful, the simplest example of an arithmetic evaluation is to alert if more then 10000 flows have been seen by pipeline in an hour. Use the RECORD COUNT primitive and the filter all from above to allow all flows through to the eval

EVALUATION simpleCount
    FILTER all
    SEVERITY 1
    ALERT ALWAYS
    ALERT EVERYTHING
    CLEAR ALWAYS
    CHECK THRESHOLD
        RECORD COUNT >10000
        TIME WINDOW 1 HOUR
    END CHECK
END EVALUATION

The CLEAR ALWAYS line causes Pipeline to reset the record count everytime is gets over 10000

Web Server (return to top)

This example uses a pair of evaluations to identify the webservers on our network, and the destination ip addresses that are talking to those web servers most often.

The first evaluation finds web servers, then puts those IPs into a list, that is used to filter records into a second evaluation that analyzes the recipients of packets from these webservers to find who's

A web server in this example is defined as a source IP (foreach SIP) that sends data to more than 500 distinct destination IP addresses and sends a total of 1MB (1000000 bytes) of data, all in a 10 minute window. These numbers are completely made up for the sake of this example, they are not an official CERT analysis.

The findWebServers evaluation sends the full list of web server IP addresses (set up using ALERT EVERYTHING) at most once an hour. It keeps a web server in the output list for up to 1 day (set using OUTPUT_LIST SIP ). It places each web server IP address it finds into the list called webServerList. An IP address will be removed from this list, if it is removed from the output list, which happens if the IP doesn't assert itself as a web server during that one day period (86400 seconds).

It uses the filter ourNetwork which is setup to only let through traffic coming from out network.

FILTER ourNetwork
    SIP == 193.0.0.0/8 #arbitrarily chosen network
    SPORT IN LIST [80, 8080] #Yes, there are more ports, but 80 and 8080 keep it simple
END FILTER

EVALUATION findWebServers
    SEVERITY 2
    OUTPUT_TIMEOUT 1 DAY
    OUTPUT_LIST SIP webServerList
    CLEAR ALWAYS
    ALERT EVERYTHING
    ALERT 1 TIMES 1 HOUR
    FILTER ourNetwork
    FOREACH SIP
    CHECK THRESHOLD
        DISTINCT DIP > 250
        TIME_WINDOW 1 HOUR
    END CHECK
    CHECK THRESHOLD
        SUM BYTES > 1000000
        TIME_WINDOW 10 MINUTES
    END CHECK
END EVALUATION

The findWebServers evaluation groups state by SIP, which is why is uses FOREACH SIP.

We only want to do this secondary analysis on traffic coming from our previously established web servers. As a result, the filter uses the output list created in the above evaluation. It checks to see if the SIP in the flow record is also in the webServerList. If so, the flow passes.

FILTER webServers
    SIP IN LIST webServerList
END FILTER

This evaluation is taking in all traffic sent out by our established web servers, this bins the records by destination IP address, and checks to see if any of these IP addresses have received more than 10MB of data in the last 15 minutes.

EVALUATION LIS
    SEVERITY 2
    ALERT SINCE LAST TIME
    ALERT 1 TIMES 60 SECONDS
    CLEAR ALWAYS
    FILTER webServers
    FOREACH DIP
    CHECK THRESHOLD
        SUM BYTES > 10000000
        TIME_WINDOW 15 MINUTES
    END CHECK
END EVALUATION

We know the SIP for all of this traffic is from a web server, so we group state by DIP, as we are identifying interesting IPs getting traffic from our web servers. This evaluation will send an alert at most once per minute, if there is something to include in an alert. When it does alert, it sends out all DIPs that hit the threshold since the last time alerts were sent because ALERT SINCE LAST TIME is used.

Monitor TCP traffic (return to top)

In another contrived example, we want pipeline to send an alert if the proportion of all traffic that is TCP goes over 75% within any 5 minute window. We use the all filter defined above because we want all traffic to go to this evaluation.

EVALUATION tcpTraffic
    SEVERITY 5
    ALERT JUST_NEW_THIS_TIME
    ALERT 1 TIMES 0 SECONDS
    CLEAR NEVER
    FILTER all
    CHECK THRESHOLD
        PROPORTION PROTOCOL 6 >= 75 PERCENT
        TIME_WINDOW 5 MINUTES
    END CHECK
END EVALUATION

Filtering Naive Scans (return to top)

Scanning-related network traffic is pervasive in most networks. While some scans may precede adverse events, for many analyses, scan traffic just adds noise and causes false positives. You can use simple filters and evaluations to create lists of scanners and conduct some of the more naive scans. Then, you can use these lists to filter scan traffic from other analyses.

Single-Packet Scanners

This filter and evaluation will create a list of IP addresses that are sending single-packet flows to a large number of unique destinations in a short time period. IP addresses will stay in the list for 10 minutes after they terminate scanning.

FILTER potentialScanFlows
    PACKETS == 1
END FILTER
EVALUATION scanners
    FILTER potentialScanFlows
    FOREACH SIP
        CHECK THRESHOLD
            DISTINCT DIP > 100
            TIME_WINDOW 5 MINUTES
        END CHECK
    OUTPUT_TIMEOUT 10 MINUTES
    OUTPUT LIST SIP scannerList_SinglePacket
    DO NOT ALERT
    CLEAR ALWAYS
END EVALUATION

To create an IP set of these addresses, use a list configuration set to overwrite a seeded set file. The initial set file can be an empty set or a set of already-known scanners.

LIST CONFIGURATION scannerList_SinglePacket
    SEVERITY 5
    SEED "scannerList_SinglePacket.set"
    OVERWRITE ON UPDATE
    UPDATE 5 MINUTES
END LIST CONFIGURATION

You can use the list to remove scan-related flows in other filters through the ANY IP NOT_IN_LIST command:

ANY IP NOT_IN_LIST scannerList_SinglePacket

TCP SYN/ACK Scanners

This filter and evaluation will create a list of IP addresses that are engaging in TCP SYN scans, where the only flag sent in a flow is the SYN flag (no acknowledgment). This will capture scans where the initial SYN is sent more than once but no other packets occur, a common scenario missed when looking only for single-packet scanners. The same can be said for ACK flows, which can be filtered by changing "FLAGS == S" to "FLAGS == A". IP addresses will stay in the list for 10 minutes after they terminate scanning.

FILTER tcpSynScan
    PROTOCOL == 6
    FLAGS == S
END FILTER
EVALUATION tcpSynScanEval
    FILTER tcpSynScan
    FOREACH SIP
        CHECK THRESHOLD
            RECORD COUNT > 100
            TIME_WINDOW 5 MINUTES
        END CHECK
    SEVERITY 2
    OUTPUT_TIMEOUT 10 MINUTES
    OUTPUT LIST SIP scannerList_tcpSynScan
    CLEAR ALWAYS
    DO NOT ALERT
END EVALUATION

To create an IP set of these addresses, use a list configuration set to overwrite a seeded set file. The initial set file can be an empty set or a set of already-known scanners.

LIST CONFIGURATION scannerList_tcpSynScan
    SEVERITY 5
    SEED "scannerList_tcpSynScan.set"
    OVERWRITE ON UPDATE
    UPDATE 5 MINUTES
END LIST CONFIGURATION

You can use the list to remove scan-related flows in other filters through the ANY IP NOT_IN_LIST command:

ANY IP NOT_IN_LIST scannerList_tcpSynScan

IPv6 Tunnel Detection (return to top)

There are three types of IPv6 automatic tunneling: Teredo, 6to4, and ISATAP. The details of each tunneling are slightly different, but their general format is the same. To detect all of them, there will be three distinct sets of filters, internal filters, and evaluations.

IPv6 tunneling happens in 2 stages:

1. Using IPv4, a node identifies a global translation server, that changes IPv4 to IPv6 and back, on known IPv4 address(es), that does the setup for the subsequent tunneling.
2. Traffic flows between the two IPv4 IP addresses with IPv6 traffic tunneled. This tunnel traffic is identified by port and/or protocol numbers.

The detection of IPv6 tunnels takes three stages:

1. Filter on the destination IP address to see if any SIP is communicating with the known IPv6 intialization servers. Put these SIPs in the list to be used by the next stage.
2. Look for traffic using a filter whose source IP is in the list created by the first stage, plus the appropriate DPORT and/or PROTOCOL combination.
3. All traffic identified by combining the first two stages is marked as tunnel traffic with no other computations, so the EVERYTHING PASSES primitive is used for the evaluation to pass all of the traffic directly into outputs and alerts.

Teredo

Filter looking for initial public server connection<:/p>

FILTER teredoInitialConnections
    DIP IN_LIST [1.2.3.4, 2.3.4.5] #Insert public ipv6 server
END FILTER

Take the flows that connecto to the ipv6 server, and store their SIPs in the list teredoSIPS, which have a timeout value of 1 hour

INTERNAL FILTER buildTeredoList
    FILTER teredoInitialConnections
    SIP teredoSIPS 1 HOUR
END INTERNAL FILTER

Look for traffic with SIP in the list, protocol UDP, DPORT 3544

FILTER teredoTraffic
    SIP IN LIST teredoSIPS
    DPORT == 3544
    PROTOCOL == 17
END FILTER

The comparisons are in that order based on the liklihood of the comparison to fail upon hitting, thus stopping the processing

Lastly, we need a basic evaluation that passes all traffic found by the teredoTraffic filter onto the alerting stage of pipeline.

EVALUATION teredo
    CLEAR ALWAYS
    SEVERITY 4
    ALERT EVERYTHING
    ALERT 1 TIMES 0 SECONDS
    FILTER teredoTraffic
    CHECK EVERYTHING PASSES
    END CHECK
END EVALUATION

6to4

6 to 4 is identified by protocol 41 after contacting the public server. The second filter in the 6to4 case is:

FILTER 6to4Traffic
    SIP IN LIST 6to4SIPS #created similarly to the first example, using a filter looking for public servers and an internal filter
    PROTOCOL == 41
END FILTER

ISATAP

ISATAP is identified the same way as 6to4, using protocol 41, but the list of public servers to initiate connection is dynamic, unlike 6to4, and teredo. As a result, the list of IPs cannot be directly typed into the filter, so an IPSet should be used, which is updated outside of Pipeline.

Sensor Outage (return to top)

The SENSOR OUTAGE file evaluation is used to send an alert if a sensor does not send flows to Pipeline within a given time period. It can monitor a list of specified sensor names, or monitor all sensors included in the SiLK sensor.conf file. This evaluation type does not look at the contents of the files (flows), so it does not need to have a filter associated with it.

The examples below send alerts if a monitored sensor does not send traffic for 2 hours.

All Sensors

FILE_EVALUATION sensorOutage
    CHECK FILE_OUTAGE
        SENSOR_LIST ALL SENSORS
        TIME_WINDOW 2 HOURS
    END CHECK
END FILE_EVALUATION

Sensor list

FILE_EVALUATION sensorOutage
    CHECK FILE_OUTAGE
        SENSOR_LIST [S1, S2, S3]
        TIME_WINDOW 2 HOURS
    END CHECK
END FILE_EVALUATION

Passive FTP(return to top)

Protocols such as FTP can use additional connections to do their data transfers. They usually use ephemeral ports that are deteremined at the time of the original standard port connection. These passive connections may be of interest. The problem is that the flows of the transfer finish first (thus they hit pipeline first), before the original port 21 connection. As a result, to detect IP addresses engaged in passive connections, we record the IP addresses from all high port to high port connections, then compare that list to flows using port 21. First we need to collect high port flows, then put the IP address in a list using an INTERNAL FILTER, then use the HIGH PORT CHECK primitive to alert on the IP addresses.

FILTER HighPorts
    SPORT >= 50000
    DPORT >= 50000
    PROTOCOL == 6
END FILTER

Put the IPs from this HighPorts filter into a specific HIGH PORT LIST using an internal filter.

INTERNAL_FILTER highPortsInternal
    FILTER HighPorts
    HIGH_PORT_LIST highPortIPs 15 MINUTES
END INTERNAL_FILTER

Now look for the original port 21 (in the case of FTP) connection

FILTER port21
    DPORT == 21
    PROTOCOL == 6
END FILTER

Use the high port primitive to compare port 21 connection IPs to the HIGH PORT LIST contents created from high port connections

EVALUATION passiveFTP
    SEVERITY 2
    OUTPUT_TIMEOUT 60 SECONDS
    ALERT SINCE_LAST_TIME
    ALERT 1 TIMES 60 SECONDS
    CLEAR ALWAYS
    INTERNAL_FILTER highPortsInternal
    FILTER port21
    FOREACH SIP DIP SPORT DPORT PROTOCOL
    CHECK HIGH_PORT_CHECK
        LIST highPortIPs
    END CHECK
END EVALUATION

Beacon Detection(return to top)

EVALUATION beacon
    CLEAR NEVER
    SEVERITY 3
    ALERT JUST_NEW_THIS_TIME
    ALERT 1 TIMES 60 SECONDS
    FILTER all
    CHECK BEACON
        COUNT 5 CHECK_TOLERANCE 5 PERCENT
        TIME_WINDOW 300 SECONDS
    END CHECK
END EVALUATION