sensie package

Module contents

class sensie.Probe(model, predict_function=None)

Bases: object

A class that wraps a pre-trained model and provides methods for testing its robustness and sensitivity to various properties.

plot_property(test, label='property', show_fit=False, fit='line', save_to=None, ticklabels=None, errorbars=True, fitorder=2)

Generates a plot from a SingleTest result.

test: SingleTest
The test to visualize.
label: str
Readable description for the property tested.
show_fit: bool
If True, a fit to the data will be plotted.
fit: str
“line” or “polynomial” - the fit to be shown.
fitorder:
For a polynomial, the order of the fit.
save_to: str
Filename to save the figure to.
ticklabels: list
Labels for the x axis. Useful (for instance) when plotting class names.
errorbars: bool
Plot error bars - one standard deviation from the mean score in the correct class.
predict_and_measure(x_test, y_test, p_test, prop=None, continuous=False, bins=20, label=None, plot=False, propnames=None, batch_size=256) → sensie.SensitivityMeasure

Scores the provided x_test and returns a SensitivityMeasure object with measured values and for plotting.

x_test: numpy.ndarray
Tensor of examples for testing
y_test: numpy.ndarray
Vector of ground-truth classes
p_test: numpy.ndarray or pandas.DataFrame
Tensor or DataFrame containing the property/properties for testing.
prop: int or str
(Optional) A numerical or string index into p_test, returning a vector or Series of the property in question. If this is None, will attempt for all columns in p_test
continuous: bool
If true, assumes the p value is continues and needs to be binned.
bins: int
Number of bins; used if continuous == True.
label: str
(Optional) An string label for the property/properties in question; used for plotting.
plot: bool
If True, produce and display a plot of the results.
propnames: list or array
A list of property names, corresponding to p_test.
batch_size: int
When calling the predict method, the batch size to use.
SensitivityMeasure
An object containing summary information about the analysis.
predict_and_measure_perturbed(x_test, y_test, perturber, p_values=None, p_min=0, p_max=1, steps=10, label=None, plot=False, ci=False, batch_size=1024) → sensie.SensitivityMeasure

Scores the provided x_test as altered by the supplied perturber function, and returns a SensitivityMeasure object with measured values and for plotting.

x_test: numpy.ndarray
Tensor of examples for testing
y_test: numpy.ndarray
Vector of ground-truth classes
perturber: function
A function, f(x_test, p), which alters (perturbs) the test set by an amount or scale p.
p_values: list or ndarray
An iterable list of p_values to be passed to the perturber function and measured. If not supplied, numpy.linspace(p_low, p_high, steps) will be used instead.
p_min: int
The minimum, and first, value for p to be passed to the perturber function.
p_max: int
The maximum, and last, value for p to be passed to the perturber function.
steps:
The number of steps from p_min to p_max to be passed to the perturber function.
label: str
(Optional) An string label for the property/properties in question; used for plotting.
plot: bool
If True, produce and display a plot of the results.
ci: bool
If True, will conduct linear fit and generate credible intervals.
batch_size: int
The x_test examples will be perturbed and scored in batches of this size.
SensitivityMeasure
an object containing summary information about the analysis.
test_class_sensitivity(x_test, y_test, plot=False)

Same as predict_and_measure, except the property is the ground truth class itself. Useful to see if certain classes in the test set have markedly different performance to others.

x_test: numpy.ndarray
Tensor of examples for testing
y_test: numpy.ndarray
Vector of ground-truth classes
plot: bool
If True, generates a plot of the results.
class sensie.SensitivityMeasure(x_test, y_test, rightscores)

Bases: object

This object wraps the individual tests performed on a model, and provides convience methods for setting credible intervals and displaying a summary.

set_credible_intervals()

Calculates credible intervals for each test performed so far (i.e. for each SingleTest instance).

summary()

Produces a summary table (as a pandas DataFrame) with the results, and significance of, tests performed. Returns:

A pandas DataFrame with a row for each test performed.
class sensie.SingleTest(property_name, p_vals, means, stds, p_points=None, y_vals=None)

Bases: object

Encapsulates the results of a single signficance test.

get_gradient() → float

Returns the gradient of the test - the change in mean score by p.

Returns: float
The gradient from a linear fit to xs, ys.
get_significance(significance_floor=0.02)

Returns a string indicating the significance of a sensitivity measure (“low”, “medium”, or “high”)

set_credible_interval(means_only=False, tune=None, samples=400)

Runs pymc3 inference code to determine the slope of the relationship between p and accuracy, and saves 50% and 95% credible intervals in instance variables. The results are stored in this SingleTest instance.

sort_and_reorder(labels=None)

Reorders the test results by y-value, i.e. the mean correct-class score. Useful for testing of discrete, unordered properties such as class.

labels: list
Labels for the classes/discrete values.
Returns: list
The provided labels, in the re-ordered order (for plotting, etc).
summary()

Show the result (gradient) of score sensitivity to this property, optionally with credible intervals.

Returns: A pandas.DataFrame with the results of the test, including credible intervals if calculated.

sensie.progbar(current, to, width=40, show=True, message=None, stderr=False)

Displays a progress bar for use in certain testing operations.