Storage and Retrieval

Experiments

The Experiment class that is used by hermes to store results also provides methods which allow you to easily access the saved data in a structured manner.

class pyhermes.storage.Experiment(identifier, path_to_experiment, commit_hash, start_time)

Experiment class mainly taking care of storage. Every instance of the class will be unqiuely associated with a specific experiment directory and a script that was executed as part of the experiment.

Experiment instances are used for the storage of important information, e.g. when it was executed, what were the command line arguments passed to the experiment script, what kind of data/results were generated during experiment execution, (…).

It also provides ways of loading experiments from disk and methods for convenient access to the experiment information.

copy_file(path_to_file, new_file_name=None)

Method to copy a file to the unique experiment directory.

Parameters

  • path_to_file – A string or path-like object

  • new_file_name – Optional new name for the copy. If None, the basename of path_to_file is used.

property exec_args

A set of the execution arguments that were stored for this experiment, e.g. {“-lr 0.001”, “–gradclip”}

Returns

The set of execution arguments

Return type

set[str]

property exec_args_dict

A dictionary mapping execution argument names to values as specified in the execution file used to run this experiment.

Returns

The execution argument dictionay

Return type

dict[str, Any]

file(file)
Parameters

file (Union[str, os.PathLike]) – Name of a file

Returns

the path to the file in the experiment directory

Return type

pathlib.Path

get_identifier()
Returns

the unique hermes identifier assigned to this experiment

Return type

str

property info_dict

Similar to read_info(), but as instance property rather than staticmethod

Returns

experiment info

Return type

dict[str, str]

static is_experiment(path)

Check if a given directory represents a hermes experiment.

Parameters

path (Union[str, os.PathLike]) – path to the directory

Returns

whether the path represents an experiment

Return type

bool

list_pth_files()

Return a list of all .pth files (i.e. pytorch state dictionaries) within the experiment directory

Returns

the list

Return type

list[str]

classmethod load(path_to_experiment)

Create an Experiment object for a finished experiment

Parameters

path_to_experiment (Union[str, os.PathLike]) –

static load_all(path)

Iterate over all subdirectories of a given parent directory and load all experiments from present experiment directories

Parameters

path – The parent directory.

Returns

A list of experiment

Return type

ExperimentList

load_state_dict(pth_name, net=None)

Load a state dictionary

Parameters

  • pth_name (str) – .pth state dict file name

  • net (torch.nn.Module, optional) – optionally load state dict into this network module

move_file(path_to_file, new_file_name=None)

Method to move a file to the unique epxeriment directory.

Parameters

  • path_to_file – A string or path-like object

  • new_file_name – Optional new name for the moved file. If None, the basename of path_to_file is used.

open_log_files(mode='r+')

Context manager for opening the STDOUT and STDERR log files.

Usage:

from pyhermes.storage import Experiment
exp = Experiment.load(...)
with exp.open_log_files(mode='r') as f_stdout, f_stderr:
   # do stuff
Parameters

mode – The filemode e.g. ‘w’, ‘a’, … Defaults to ‘r+’.

open_stderr_file(mode='r')

Context manager for opening the STDERR file only

Parameters

mode – The filemode e.g. ‘w’, ‘a’, … Defaults to ‘r+’.

static read_info(path_to_experiment)

Try to read the hermes.info of a hermes experiment directory and return it as dictionary

Parameters

path_to_experiment (Union[str, os.PathLike]) – path to the experiment directory

Returns

experiment info

Return type

dict[str, str]

property result

Load the pyhermes result associated with this experiment, if it exists :return: pyhermes.result.Result :raises RuntimeError: if the experiment does not contain a pyhermes result

scores()

Loads the scores from the experiments’ scores file

Returns

The scores

Return type

numpy.ndarray

property scores_file

Returns the path to the scores file stored in this experiment as a pathlib.Path

store_comments()

Store comments given by -c option either through main call or hermes option :meta private:

Experiment Lists

When loading multiple experiments with pyhermes.storage.Experiment.load_all(), hermes returns a list of experiments.

class pyhermes.storage.ExperimentList(exp_iterable)

A wrapper around list. Offers some useful functions for selecting/filtering loaded experiments. Will be extended in the future.

These lists are typically created by pyhermes.storage.Experiment.load_all(), but you can also create them in your own.

filter_by_exec_args(*args)
Parameters

args – Argument as strings

Returns

A new experiment list containing only those experiments which have the given args in their exec args set.

Usage example:

from pyhermes.Storage import Experiment
path = "some path where many experiments are stored"
experiments = Experiment.load_all(path)
ddqn_runs = Experiment.filter_by_exec_args("--doubledqn")
latest()

This method makes use of the fact hermes tracks the start time of each experiment,

Returns

The most recent experiment within the list

Return type

Experiment

Experiment Frames

If you installed pandas as optional dependency, you can also transform an ExperimentList into a special pandas.DataFrame that allows even more flexible access to experiments via their exec arguments.

pyhermes.storage.ExperimentFrame

alias of pyhermes.storage._ExperimentFrame

Results

Hermes provides a wrapper-class around python dictionaries that tries to simplify system IO-operations, dictionary write- and read-access and also has a rudimentary plot functionality

class pyhermes.result.Result(name, path=None, dictionary=None)

A class for storing results, i.e. data that is generated during the execution of some script/experiment.

This class inherits from dict and provides useful methods for - storing data in the underlying dictionary - saving the dictionary to disk - accessing stored data - plotting stored data

Note that these plots are meant for rapid prototyping/debugging and are neither very pretty, nor camera-ready as of yet.

classmethod from_namespace(name, namespace)

Create a new result from an existing namespace

Parameters

  • name (str) – the result name

  • namespace (namespace) – the namespace to use

classmethod load(path)

Load a result from disk

Parameters

path – the path to a result file

Some things you should know about this method: - stored scalar lists are converted to numpy arrays on load - internally it uses torch.load

Returns

the result

classmethod new(name)

Create a new result and give it a name

Parameters

name (str) – the result name

Returns

the result

Return type

Result

plot_scalar(key, smoothing=50, show=True, ax=None)

Plot the array of scalars stored under a specific key

Parameters

  • key (str) – the scalars key

  • smoothing (int, optional) – Size of a smoothing window applied to the scalars before plotting, defaults to 50

  • show (bool, optional) – Whether to immediately show the plot, i.e. call plt.show, defaults to True

  • ax (matplotlib.axis, optional) – An axis to to plot on, defaults to None

save(directory=None)

Save the result to disk. Uses torch.save internally.

Parameters

directory – the directory, where to save the result. Set to None, if the result already has a path and you wish to overwrite,

default to None. Note that you will have to specify a path on every first save. :type directory: Union[str, os.PathLike]

store_array(value, key)

Similar to store_scalar(), but store lists of values, without flatting them into one dimension. After loading this list, it will be a 2d numpy :type:numpy.ndarray.

Parameters

  • value (Union[List[Any], numpy.ndarray]) – The array to store

  • key (str) – The key under which the array is stored

store_scalar(value, key)

Store a scalar, or list/iterable of scalars associated with some key, i.e. name / string identifier.

Parameters

  • value (Union[bool, int, float, Iterable[Union[bool, int, float]]]) – The value to store

  • key (str) – The key under which the scalar is stored

If the key is new, a list entry in the result dictionary is associated with it, otherwise the scalar(s) is (are) appended to the key’s list.

Evaluation Utilities

Plotting

The module pyhermes.plot provides you with a multitude of functions which can aid you with visualizing experiment results such as training curves.

pyhermes.plot.plot_training_progress(scores, eps_start=None, eps_end=None, eps_decay=None, epsilons=None, window_length=250, figsize=(12, 12), fontsize=18, path=None, scores_color='c', mean_color='r', score_range=None, epsilon_range=None, axis_gap=0.02, mark_best=False, legend=True, ax=None)

Plot the training curve given an array of training scores, i.e. training episode returns

Parameters

  • scores (float array) – scores, i.e. training episode returns

  • eps_start (int, optional) – epsilon-greedy exploration coefficient in the training beginning. Defaults to 1.

  • eps_end (float, optional) – epsilon-greedy exploration coefficient in the end of training. Defaults to 0.001.

  • eps_decay (float, optional) – epsilon-greedy exponential decay factor from eps_start to eps_end. Defaults to 0.999.

  • epsilons (list, optional) – alternative to eps_start, eps_end and eps_decay. Provide all used eps values. Defaults to []. Will be ignored, if eps_start, eps_end and eps_decay are given

  • window_length (int, optional) – size of window_length for sliding window. Defaults to 250.

  • figsize (tuple, optional) – Figure size to use.. Defaults to (12,12).

  • fontsize (int, optional) – fontsize to use. Defaults to 18.

  • path (string, optional) – path to save the created plot. Defaults to None.

  • scores_color (str, optional) – score dot color. Defaults to ‘c’.

  • mean_color (str, optional) – slidign mean color. Defaults to ‘r’.

  • score_range ((float, float), optional) – range to be shown in the plot on the y-axis. Defaults to None.

  • epsilon_range ((float, float)), optional) – range to be shown in the plot in the epsilon y-axis. Defaults to None.

  • axis_gap (float, optional) – gap to be added to score_range and epsilon_range for better readability. Defaults to 0.02.

  • mark_best (bool, optional) – whether the best observed value should be marked. Defaults to False.

  • legend (bool, optional) – whether to plot a legend. Defaults to True.

  • ax (pyplot axis, optional) – pyplot axis to use. Generating a new one if none provided. Defaults to None.

Returns

generated plot.

Return type

plt

pyhermes.plot.plot_training_progress_from_experiment(exp, eps_start_key='-es', eps_end_key='-ee', eps_decay_key='-ed', window_length=250, figsize=(12, 12), fontsize=18, path=None, scores_color='c', mean_color='r', score_range=None, epsilon_range=None, axis_gap=0.02, mark_best=True, legend=True)

Read the trainings scores from file and plot the training progress.

Parameters

  • exp (Experiment) – experiment

  • window_length (int, optional) – size of window_length for sliding window. Defaults to 250.

  • figsize (tuple, optional) – Figure size to use.. Defaults to (12,12).

  • fontsize (int, optional) – fontsize to use. Defaults to 18.

  • path (string, optional) – path to save the created plot. Defaults to None.

  • scores_color (str, optional) – score dot color. Defaults to ‘c’.

  • mean_color (str, optional) – slidign mean color. Defaults to ‘r’.

  • score_range ((float, float), optional) – range to be shown in the plot on the y-axis. Defaults to None.

  • epsilon_range ((float, float)), optional) – range to be shown in the plot in the epsilon y-axis. Defaults to None.

  • axis_gap (float, optional) – gap to be added to score_range and epsilon_range for better readability. Defaults to 0.02.

  • mark_best (bool, optional) – whether the best observed value should be marked. Defaults to False.

  • legend (bool, optional) – whether to plot a legend. Defaults to True.

Returns

Plot

Return type

pyplot

pyhermes.plot.plot_training_progress_from_file(file, eps_start=1, eps_end=0.001, eps_decay=0.999, epsilons=[], window_length=250, figsize=(12, 12), fontsize=18, path=None, scores_color='c', mean_color='r', score_range=None, epsilon_range=None, axis_gap=0.02, mark_best=True, legend=True)

Read the trainings scores from file and plot the training progress.

Parameters

  • file (string) – path to file

  • eps_start (int, optional) – exploration coefficient in the training beginning. Defaults to 1.

  • eps_end (float, optional) – exploration coefficient in the end of training. Defaults to 0.001.

  • eps_decay (float, optional) – exponential decay factor from eps_start to eps_end. Defaults to 0.999.

  • epsilons (list, optional) – alternative to eps_start, eps_end and eps_decay. Provide all used eps values. Defaults to [].

  • window_length (int, optional) – size of window_length for sliding window. Defaults to 250.

  • figsize (tuple, optional) – Figure size to use.. Defaults to (12,12).

  • fontsize (int, optional) – fontsize to use. Defaults to 18.

  • path (string, optional) – path to save the created plot. Defaults to None.

  • scores_color (str, optional) – score dot color. Defaults to ‘c’.

  • mean_color (str, optional) – slidign mean color. Defaults to ‘r’.

  • score_range ((float, float), optional) – range to be shown in the plot on the y-axis. Defaults to None.

  • epsilon_range ((float, float)), optional) – range to be shown in the plot in the epsilon y-axis. Defaults to None.

  • axis_gap (float, optional) – gap to be added to score_range and epsilon_range for better readability. Defaults to 0.02.

  • mark_best (bool, optional) – whether the best observed value should be marked. Defaults to False.

  • legend (bool, optional) – whether to plot a legend. Defaults to True.

Returns

Plot

Return type

pyplot

pyhermes.plot.smoothen(arr, window_size=100)
Function that computes the sliding mean over the given array.

For the first numbers n < window_size, the sliding mean is computed using the numbers there are.

Parameters

  • arr (float array) – scores

  • window_size (int, optional) – size of sliding window. Defaults to 100.

Returns

list with sliding means, list with stds

Return type

(float array, float arrary)