Creating and using a Dataset object¶

Dataset objects are created from the Toolbox object. The new_dataset_from_filename and new_dataset_from_stream methods create datasets from external data like files or network streams. The new_dataset_from_columns and new_dataset_from_rows methods create datasets from data in memory. To create an empty Dataset object, pass an empty list into either the new_dataset_from_columns or new_dataset_from_rows method.

After a Dataset object has been instantiated, new columns can be added to it with the append_column method. Therefore, this:

# t is a Toolbox object
# c1, c2, and c3 are Column objects
d = t.new_dataset_from_columns([c1, c2, c3])

is equivalent to this:

d = t.new_dataset_from_columns([])
d.append_column(c1)
d.append_column(c2)
d.append_column(c3)

Headers can be added with the set_header method:

# Add header named "foo" with value "bar"
d.set_header("foo", "bar")

Column names can be added with the set_column_names method:

# d is a dataset object with 3 columns
d.set_column_names(["foo", "bar", "baz"])

Comparing Dataset objects¶

Dataset objects may be compared for equality. Two Dataset objects are considered equal to each other if and only if:

• They contain an equal number of columns
• Their lists of column names are identical (same length, contents and order).
• They have equal columns in the same order. Column equality is determined by comparing each Column object.

Accessing Data in Dataset Objects¶

The Dataset object supports row-major and column-major iteration. The default iterator is row-major; for r in d iterates over all the rows in dataset d in the current sort order. To iterate over the rows in native sort order (regardless of the current sort order), use Dataset.iter_ignore_sorted. To get an iterator over all the columns in a Dataset object, use Dataset.iter_columns.

The Dataset object does not support array-based direct access: d[0] will raise an error. To refer to a specific column or row in a dataset, use the Dataset.get_column or Dataset.get_row methods. To refer to a specific field in a dataset, get the appropriate row or column and access the field directly:

# d is a dataset with 10 rows
# and columns foo, bar and baz

# Gets value of column foo in 5th row
d.get_row(4).foo
# ...or...
d.get_column('foo')[4]

get_column and get_row return Column and Row objects. See Columns and Rows for more information on these objects.

Sorting and Filtering Dataset objects¶

It is often convenient to work with data that is sorted in different ways, or with subsets of data. Sometimes, as with top n lists, we want to do both.

Datasets can be sorted using the Dataset.sort method. This method takes a key function which takes a Row as input, and returns a comparable value which is actually used in the sort. The Dataset.sort method changes the sort order of the Dataset object, which changes the order through which the data is iterated or accessed through the get_row method. It does not change the ordering of data on-disk; the native sort order (or insert order) can always be recovered by calling the Dataset.sort method with no arguments.

The Dataset object’s filtering methods also take a function. The filtering function takes a Row object as input, and returns True if the row passes the filter, False if it fails. (As a shortcut, True is equivalent to lambda r: True and False is equivalent to lambda r: False; these are sometimes used in conjunction with the limit parameter to select a portion of the dataset.) The Dataset.filter_pass and Dataset.filter_fail methods are utility methods to simplify invocations to Dataset.filter; if both the passing and failing sets of data are of interest, they can be captured in a single pass through the data using Dataset.filter_both.

To generate a top n list from a dataset, first sort the data on the desired column, then filter it with a limit parameter of n:

# Get the top n elements of d, sorted by num_hits
d.sort(key=lambda r: r.num_hits)
top_n_by_num_hits = d.filter_pass(True, limit=n)

Exporting Dataset objects¶

Dataset objects can be exported to disk, streams or strings in the format described in On-disk format. The methods for this are analogous to those described in Creating and using a Dataset object.

>>> d.to_file("bar.txt")
>>> ostrm = open("bar2.txt", 'w')
>>> d.to_stream(ostrm)
>>> print d3.to_string()
1|a
2|b
3|c

On-disk format¶

This section describes the format Rayon uses for data import and export. This format is fundamentally delimited text, with additional facilities for storing metadata with the dataset. The SiLK tools’ text output (if --no-titles and --no-columns are passed to the tool as options) is a subset of this format, and may be passed to Rayon unchanged.

Rayon’s default delimiter is the pipe character (|). The delimiting character must not appear in the input. An example of valid input is:

foo|1|2
bar|3|4

Whitespace around delimiters will be removed, so the following is equivalent to the above:

foo| 1| 2
bar| 3| 4

Whitespace lines are also ignored, so this is equivalent to the above:

foo|1|2

bar|3|4

The Rayon data format does not support infix or postfix comments. In the following, the text # this is not will be interpreted as content:

# this is a comment
foo | a | a
bar | b | b # this is not

foo| 1| 2
# this line will be ignored
bar| 3| 4

## Description: This is an example dataset
foo| 1| 2
bar| 3| 4

## Delimiter: ,
foo,1,2
bar,3,4

## Column-Names: label|value1|value2
foo|1|2
bar|3|4

# label| value1| value2
foo|1|2
bar|3|4

#label| value1| value2
foo|1|2
bar|3|4

#label|value1|value2
foo|1|2
bar|3|4

Comparing Column objects¶

Column objects can be compared for equality. Two Column objects are equal if each item in one column compares as equal to the item in the other column at the corresponding index.

Accessing Data in Column objects¶

Column objects can be treated like arrays:

>>> from rayon.toolbox import Toolbox
>>> t = Toolbox.for_file()
>>> raw = [[1,2,3], ['a', 'b', 'c']]
>>> d = t.new_dataset_from_columns(raw, colnames=['foo', 'bar'])
>>> c = d.get_column('bar')
>>> c[0]
'a'
>>> len(c)
3
>>> list(c)
['a', 'b', 'c']

However, Column objects are immutable, so this won’t work:

>>> c[0] = 'z'
...
TypeError: 'Column' object does not support item assignment

Column Statistics¶

Column objects provide several statistical functions on the data. With the exception of uniq, these methods require that the Column object contain numeric data.

The following functions are available:

# c is a Column object
c.max()             # Largest value
c.mean()            # Average value
c.min()             # Smallest value in column
c.percentile(25)    # 25th percentile value
c.sample_stdev()    # Sample standard deviation
c.sample_variance() # Sample variance
c.stdev()           # Population standard deviation
c.variance()        # Population variance
c.uniq()            # List of all unique values

In general, if a statistical function is available from the Column object, it is better to use it than to compute it independently because the Column object will cache values and (where necessary) sorted order.