A Note on “Unexpected”

Our notion of unexpected is tied to our experience of what happens and what doesn’t happen, or the way that certain things happen. It is tied to our notions of what is common.

Common is what has precedent (what happens) and some level of regularity and/or consistency - some phenomenon which occurs within somewhat predictable bounds of frequency, volume, time, etc. It is generally expected that a phenomenon continues to occur within or around those bounds. It is generally unexpected when the phenomenon occurs too far outside of those bounds or when a sufficiently new phenomenon occurs. There’s a difference between “commonly occurring” and “commonly occurring this way.” “Commonly occurring this way” can also be thought of as consistency. Think about your expectations and level of surprise by the sun rising each day at around the same time or the cycles of the moon.

We apply these concepts to our outbound views of certain network traffic. We track how frequently we observe certain flow characteristics (as a percentage of days seen in the last X days of baseline network traffic), the volume at which they tend to occur (average flow count per day seen), and when they tend to occur (days of the week, hours of the day). Also, we track application, packets, bytes, and duration tendencies per those same flow characteristics.

Percent days seen is the best reflection of “commonly occurring.” All other measurements are more a reflection of “commonly occurring this way”/consistency.

Unexpected outbound protocol use occurs when certain flow characteristics are observed but which have never been seen before, or when certain flow characteristics are observed and have been seen before, but which have only rarely occurred or are sufficiently inconsistent with past occurrences.

So what are the flow characteristics?

The Anchor Flow Characteristics

A Quick Note on Levels of Specifity (from least to most specific)

Source: parent organization → organization → sensor → src-net-block → sip

Destination: ASN info (number, country code, registry, org) → dst-net-block → dip → dport → protocol/service

From left to right, each pair typically has a one-to-many relationship. For example:

Importantly, observed outbound dports ultimately depend on what is allowed to egress a network. The observed dports may or may not change often - this is more a reflection of the velocity, fluidity, or lack of egress controls of each source organization/enclave. Ideally, they will be tightly controlled, and you would expect certain outbound protocols to have more stringent rules regarding their use. Either way, we will know about it.

In order to balance efficacy and alert generation, we anchor/scope protocol baseline traffic measurements to different levels of specificity – the Full Anchor Tuple and the Partial Anchor Tuple, which essentially act as unique keys for the calculation and aggregation of other flow characteristics.

Full Anchor Tuple (FAT)

Consists of: src-org|sensor|sip|dip|dport|asn_info where asn_info → dst-netblk|asn|cc|rir|org

This is the most specific set of flow characteristics on which protocol baseline traffic measurements are calculated. It essentially accounts for unique source/destination IP pairs across any combination of src-org, sensor, dport, and asn_info. These baseline entries are used for consistency checks, if a matching tuple exists for an incoming flow, as well as to add context to generated alerts.

Partial Anchor Tuple (PAT)

Consists of: sensor|dport|asn_info where asn_info → dst-netblk|asn|cc|rir|org

This is the slightly abstracted and broadest set of flow characteristics on which protocol baseline traffic measurements are calculated. IP addresses are more variable for numerous reasons (e.g., unknown/complex source network architecture, CDNs, load balancers, routing policies, etc.). So we treat variations in sip and/or dip as less significant than variations in sensor|dport|asn_info combinations. These baseline entries ultimately determine whether an incoming flow for a given protocol was NEVER_SEEN_IN_BASELINE or SEEN_BUT_RARELY_OCCURRING (see below). By default, we allow PAT matches which are above set thresholds to pass, that is, not alert.

The Measurements

Each flow of the baseline traffic is first lightly enriched with source org and destination ASN information.

The following measurements are calculated for each unique FAT and PAT:

The following additional measurements are calculated for each unique PAT:

Thresholds and Baseline Checks

There is a set of configurable thresholds for each protocol. Those include the following:

Global sane defaults for the thresholds are already set, but you may adjust as necessary or add protocol-specific thresholds via the protcol_thresholds dictionary defined in scripts/lambda_functions/check_xdeny_baseline_sal.py.

protcol_thresholds = {
  'global' : { 'perc_days_seen': 15.0, 'consistency_score': 85, 'standard_deviations': 3.0 },
  '<protocol-specific>' : { ... }
}

The Consistency Score

The consistency score starts at 100 and various deviation checks are performed. For each deviation, some number of points is deducted from the consistency score. Since this is an outbound analytic, for packets, bytes, and duration, we only care about deviations which are higher than usual - we neither check nor deduct any points for deviations which are lower than usual. We weigh deviations in applications seen and bytes more than other deviations.

These checks are not performed at all if no matching baseline entry exists for the PAT of an incoming flow. If a matching baseline entry exists for the FAT of an incoming flow and the FAT was seen on at least 2 days and total_days_seen * avg_flow_count >= 10, then the measurements associated with the FAT entry are used for these checks; otherwise, the measurements associated with the PAT entry are used.

The deviation checks are as follows:

So, with a default consistency_score of 85, the following sets of deviations are the minimum needed to generate an alert:

The Baseline Checks

Each incoming flow’s PAT is derived and looked up in the baseline database. If no matching entry exists, it is marked as an unexpected record and the alert reason is NEVER_SEEN_IN_BASELINE.

If a matching entry does exist, then the PAT entry’s perc_days_seen is compared against the perc_days_seen threshold. If the entry’s perc_days_seen is less than the set threshold, it is marked as an unexpected record and the alert reason is SEEN_BUT_RARELY_OCCURRING.

Lastly, if a matching entry exists, the consistency score is calculated according to the above section. If the calculated consistency score is less than the set threshold, the alert reason will be SEEN_BUT_INCONSISTENT, but only if the incoming flow is not considered “rarely occurring.” Either way, the consistency score is calculated and included in all SEEN_BUT_RARELY_OCCURRING and SEEN_BUT_INCONSISTENT alerts.

Explicit Deny and Allow Lists

You can add entries containing one or more rules to match on incoming flows and select metadata to one of two types of lists.

The Explicit Deny List applies prior to the baseline checks and allows you to match on flow characteristics that you want to alert on immediately. Sort of like a watchlist.

The Allow List applies after the baseline checks and only on flows marked as an unexpected record. This allows you to suppress certain flow characteristics for which you no longer wish to receive alerts for any number of reasons.

To help make the most efficient use of enrichment data sources and APIs, there are two versions of Allow List.

The Simple Allow List (SAL) is checked prior to enrichments being added to the record. If a record matches on the SAL, it gets written directly to the unx-obp-allowlisted-* index, bypassing any further enrichments or checks. Otherwise, enrichments will be added to the record.

The Enhanced Allow List (EAL) is checked after enrichments are added to the record, and as such has additional fields available with which to define rules.

Entries and rules in both types of lists follow the same basic format.

Entries

Entries may be defined on a global level (applies to all flows/protocol traffic incoming to UNX-OBP) or per protocol (applies to incoming flows for a single protocol/dataset). Each entry has the following fields:

Rules

There may be one or more rules defined in each entry. Each rule is a single string with fields/values defined as field=value separated by semicolons.

Only positive hits on all fields in a rule against the incoming flow is considered a match. No field should be present in a rule more than once. Rules can be any order/combination of one or more of the following fields:

In addition to the above fields and applicable to Allow List rules only:

In addition to the above fields and applicable to Enhanced Allow List rules only:

Ranges

For fields that support ranges, checks are always inclusive and you must specify at least one limit (lower or upper) with a hyphen. Ranges may be included in a comma-separated list of values. See the following examples with ranges:

Modifiers

For fields that support modifiers, you can add an “&” (and) and/or an “!” (only) to the end of the final value in order to employ some additional logic beyond simple OR logic. For example: