API¶
functional¶
A collection of function and variables used to define experiments using the functional api
-
class
xmen.functional.
Root
(root=None, purpose='', copy_params=True, **kwargs)[source]¶ The first argument passed to a functional experiment and principally used to root an experiment instance to a particular directory:
def functional_experiment(root: Root, ...): with open(root.directory, 'w') as f: f.write('Running experiment) root.message({'time': time.time()})
Note
Root is nothing more than Experiment with a different name. Whilst principally offering exactly the same functionality, primarily the purpose of Root is to expose the directory property and messaging protocol of the Experiment class to functional experiment definitions. However, there is nothing stopping the user form using the full functionality of the Experiment class if they wish. Please consult the Experiment class documentation in this case.
-
xmen.functional.
read_comments
(fn)[source]¶ A helper function for reading comments from the function definition. This should not be generally needed as xmen takes care of this for you.
- Parameters
fn – A functional experiment definition conforming to the xmen api
- Returns
A help string generated for each function argument.
- Return type
docs (str)
-
xmen.functional.
functional_experiment
(fn)[source]¶ Convert a functional experiment to a class definition. Generally this should not be needed as xmen takes care of this for you. Specifically:
The parameters of the experiment are added from the argument of the function
Comments next to each argument will be automatically added to the doc string of the experiment
The experiments run method will be set to
fn
- Parameters
fn (xmen.Root) – An experiment definition conforming to the xmen functioanl api (the function must take as its first argument an object inheriting from experiment)
- Returns
A class equivalent definition of fn
- Return type
Exp (class)
experiment¶
-
class
xmen.experiment.
Experiment
(root=None, purpose='', copy_params=True, **kwargs)[source]¶ Base class from which all other experiments derive. Experiments are defined by:
Parameters: class attributes declared with the special parameter
# @p
in a comment after the definition.Execution: defined by overloading the
run()
method
For example:
class AnExperiment(Experiment): ''' Doc strings should be used to document the purpose of the experiment''' # experiment parameters are defined as class attributes with an @p tag in there comment a: str = 'Hello' # @p a parameter b: str = 'World!' # @p another parameter # experiment execution code is defined in the experiments run method def run(self): print(f'{self.a} {self.b})
-
__init__
(root=None, purpose='', copy_params=True, **kwargs)[source]¶ Create a new experiment object.
- Parameters
name (root,) – If not None then the experiment will be registered to a folder
{root}\{name}
purpose (str) – An optional string giving the purpose of the experiment.
copy_params (bool) – If True then parameters are deep copied to the object instance from the class definition. Mutable attributes will no longer be shared.
**kwargs – Override parameter defaults.
-
property
start
¶ The time the experiment last started running.
-
property
registered
¶ The time the experiment was last registered.
-
property
stopped
¶ The time the experiment was last stopped.
-
property
last
¶ The time the experiment state was last communicated.
-
property
notes
¶ A dictionary containing the notes attached to the experiment
-
property
root
¶ The root directory to which the experiment belongs
-
property
status
¶ The status of the experiment. One of
'default'
,'registered'
,'running'
,'finished'
or'error'
.
-
property
created
¶ The date the experiment parameters was first registered.
-
property
purpose
¶ A string giving a purpose message for the experiment
-
property
messages
¶ A dictionary of messages logged by the experimet.
-
property
version
¶ A dictionary giving the version information for the experiment
-
property
user
¶ The current user of the experiment
-
property
host
¶ The host of the experiment
-
note
(string, remove=False)[source]¶ Leave a note with the experiment. Will be removed if
remove
isFalse
-
to_defaults
(defaults_dir)[source]¶ Create a
defaults.yml
file from experiment object. Any base class inheriting from Experiment can create a default file as:MyExperiment().to_yaml('/dir/to/defaults/root')
-
debug
()[source]¶ Inherited classes may overload debug. Used to define a set of open_socket for minimum example
-
from_yml
(path, copy=False)[source]¶ Load state from either a
params.yml
ordefaults.yml
file (inferred from the filename). The status of the experiment will be updated to'default'
if'defaults.yml'
file else'registered'
ifparams.yml
file.If copy is
True
then only user defined parameters themselves will be copied from the params.yml file.
-
register
(root, purpose='', force=True, same_names=100, restart=False, **_)[source]¶ Register an experiment to an experiment directory. Its status will be updated to
registered
. If an experiment calledname
exists inroot
andforce==True
then name will be appended with an int (eg.{name}_0
) until a unique name is found inroot
. Ifforce==False
aValueError
will be raised.If restart is also passed, and an experiment called name also exists, then the experiment will be loaded from the params.yml file found in
'{root}/{name}'
.- Raises
ValueError – if
{root}/{name}
already contains aparams.yml
file
-
to_root
(root_dir, shell='/bin/bash')[source]¶ Generate a
defaults.yml
file andscript.sh
file inroot_dir
.- Parameters
root_dir (str) – A path to the root directory in which to generate a script.sh and defaults.yml to run the experiment.
-
message
(messages, keep='latest', leader=None)[source]¶ Add a message to the experiment (and an experiments params.yml file). If the experiment is not registered to a root then no messages will be logged.
- Parameters
messages (dict) – A dictionary of messages. Keys are interpreted as subjects and values interpreted as messages. If the
defaults.yml
already contains subject then the message for subject will be updated.keep (str) – which message to keep in the case of collision. One of [‘latest’, ‘min’, ‘max’]
leader (str) – If not None then all messages will be saved if the keep condition is met for the leader key.
Note
Only messages of type float, int and string are supported. Any other message will be converted to type float (if possible) then string thereafter.
-
main
(args=None)[source]¶ - Take the command line args and execute the experiment (see
parse_args
for more information). In order to expose the command line interface:
if __name__ == '__main__': exp = AnExperiment().main()
Note that for backwards compatibility it is also possible to pass
args
as an argument tomain
. This allows the experiment to be run from the commandline as:if __name__ == '__main__': exp = AnExperiment() args = exp.parse_args() exp.main(args)
- Take the command line args and execute the experiment (see
monitor¶
Monitor¶
-
class
xmen.monitor.
Monitor
(*, hooks=[], log=None, log_fn=None, log_format='', time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶ Automate tracking and logging of experiments.
Configured through arguments of the form
f'{regex}@{modulo}{trigger}'
. Any variables in the local scope of the monitor which match the specified regular expression will be logged every modulo steps of the trigger. Triggers are incremented either manually (using theinc
method) or automatically using nested iterators (see example). Supported triggers include [“step”, “epoch”, “era”, “eon”, “supereon”] or their abbreviations [“s”, “e”, “er”, “eo”, “se”].If a hook is passed with modulo == None it can instead be triggered manually as
monitor.to_hooks()
.Example:
from xmen import Experiment from xmen.monitor import Monitor X = Experiment(..., ...) a, b, c, d, e, f = 0, 0, 0, 0, 0, 0 def identity(x): return x def mult(x): return 10 * x m = Monitor( log=('a|b@2s', 'b@1e', 'c@1er', "d|e@1eo", "e@1se"), log_fn=(mult, identity, identity, identity, identity), log_format='.3f', msg='a->X@1s', time=('@2s', '@1e'), probe='X@10s', limit='@20s') for _ in m(range(2)): for _ in m(range(2)): for _ in m(range(3)): for _ in m(range(4)): for _ in m(range(5)): a += 1 b += 1 c += 1 d += 1 e += 1
-
__init__
(*, hooks=[], log=None, log_fn=None, log_format='', time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶ - Parameters
hooks (Iterable[Hook]) – User defined hooks used to extend the functionality of the Monitor class inheriting from
Hook
.log (str, Iterable[str]) – A modulo string of the form
"f{regex}"
or"{regex}@{steps}s"
(or list of) giving the variables to log @ a particular logging frequency to stdout.log_fn (Callable, Iterable[Callable]) – Converts the variable into a string for logging.
log_format (str, Iterable[str]) – A format used to format a string as
f"{string}:{format}"
time (str, Iterable[str]) – A string of the form
f"@{steps}"
to log timing statistics at (or list of for different triggers).msg (str, Iterable[str]) – A modulo string of the form
"{regex}->{exp_regex}@{steps}s"
(or list of) giving the variables to log as messages with the experiments matchingexp_regex
.msg_keep (str, Iterable[str]) – One of [‘latest’, ‘max’, ‘min’] giving the protocol to use on message collision
msg_leader (str, Iterable[str]) – A regex to a single variable. If supplied then this variable will be treated as the leader and all other variables will be logged only if the keep condition is met for the leader
msg_expand (bool, Iterable[str]) – If True then dictionary variables with K keys will be expanded to give K variables
msg_prep (bool, Iterable[str]) – If True then each variable in the dictionary will be prepended by the dictionary name.
probe (str, Iterable[str]) – A string of the form
f"{regex}@{steps}"
to log resource use to each experiment that matches regex (or list of for different triggers).limit (str, Iterable[str]) – A modulo string of the form ``f”@{modulo}{triger}” used to limit the number of iterations of an experiment at a particular trigger level. This is useful if an experiment is restarted for example.
Note
All variables
ckpt_keep
,msg_leader
,msg_expand
andmsg_prep
can be supplied as a single or as a list of entries, one for each set of variables matching each modulo string in each case.
-
inc
(trigger, back=1)[source]¶ Manually increment trigger. Will also run modulo hooks defined by the user.
-
log
(x, format='', process_func=<class 'str'>)[source]¶ Log string with time, epoch and step.
- Parameters
x – Value to be logged
format – The format used to convert x to a string as ‘{x:{format}}’
process_func – A callable able to convert x to a valid format for printing as {x:{format}}’
-
TorchMonitor¶
-
class
xmen.monitor.
TorchMonitor
(directory=None, *, hooks=[], ckpt=None, ckpt_keep=None, log=None, log_fn=None, log_format='', img=None, img_fn=None, img_pref='', img_prep=True, img_options={}, sca=None, sca_fn=None, sca_pref='', sca_prep=True, hist=None, hist_fn=None, hist_pref='', hist_prep=True, fig=None, fig_fn=None, fig_pref='', fig_prep=True, txt=None, txt_fn=None, txt_pref='', txt_prep=True, vid=None, vid_fn=None, vid_pref='', vid_prep=True, time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶ Automate tracking, logging and saving of experiments with additional hooks for logging to tensorboard and checkpointing of experiments.
Manual logging and checkpoint are also supported as
monitor.log(...)
andmonitor.checkpoint(...)
-
__init__
(directory=None, *, hooks=[], ckpt=None, ckpt_keep=None, log=None, log_fn=None, log_format='', img=None, img_fn=None, img_pref='', img_prep=True, img_options={}, sca=None, sca_fn=None, sca_pref='', sca_prep=True, hist=None, hist_fn=None, hist_pref='', hist_prep=True, fig=None, fig_fn=None, fig_pref='', fig_prep=True, txt=None, txt_fn=None, txt_pref='', txt_prep=True, vid=None, vid_fn=None, vid_pref='', vid_prep=True, time=(), msg=None, msg_keep='latest', msg_leader=None, msg_expand=False, msg_prep=None, probe=None, limit=None)[source]¶ - Parameters
directory (str) – The directory used to log checkpoints in
hooks (Iterable[Hook]) – User defined hooks used to extend the functionality of the Monitor class
ckpt (str, Iterable[str]) – A modulo string of the form
"f{regex}"
or"{regex}@{steps}s"
(or list of) giving the variables to checkpoint @ a particular logging frequency to stdout. The regex much match objects inheriting form torch.Module.ckpt_keep (int, Iterable[int]) – The number of checkpoints to keep. The most recent checkpoints will be kept. If None then all checkpoints will be kept.
log (str, Iterable[str]) – A modulo string of the form
"f{regex}"
or"{regex}@{steps}s"
(or list of) giving the variables to log @ a particular logging frequency to stdout.log_fn (Callable, Iterable[Callable]) – Converts the variable into a string for logging.
log_format (str, Iterable[str]) – A format used to format a string as
f"{string}:{format}"
time (str, Iterable[str]) – A string of the form
f"@{steps}"
to log timing statistics at (or list of for different triggers).img (str, Iterable[str]) – A modulo string of the form
"f{regex}"
or"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard images @ a particular logging frequency.img_fn (Callable, Iterable[Callable]) – Converts the variable into an image of shape [B, C, H, W] or [C, H, W] for tensorboard. See TensorBoardLogger for more details in terms of automatic-processing. If
img
is a list thenimg_fn
can also be passed as list with a callable for each entry inimg
or can be passed as a single callable used for all entries.img_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
img_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
img – A modulo string of the form
"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard scalars @ a particular logging frequency.sca_fn (Callable, Iterable[Callable]) – Converts the variable into a scalar for tensorboard. See TensorBoardLogger for more details in terms of automatic-processing. If
sca
is a list thensca_fn
can also be passed as list with a callable for each entry insca
or can be passed as a single callable used for all entries.sca_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
sca_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
hist (str, Iterable[str]) – A modulo string of the form
"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard histograms @ a particular logging frequency. If no logging frequency is supplied then any variable logged in the experiment which matchesregex
will be logged to tensorboard each time it is passed to the logger. This is useful for logging variables at the end of an epoch for example.hist_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard. If
hist
is a list thenhist_fn
can also be passed as list with a callable for each entry inhist
or can be passed as a single callable used for all entries.hist_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
hist_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
fig (str, Iterable[str]) – A modulo string of the form or
"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard figures @ a particular logging frequency.fig_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard into a plt.figure(). If
fig
is a list thenfig_fn
can also be passed as list with a callable for each entry infig
or can be passed as a single callable used for all entries.fig_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
fig_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
txt (str) – A modulo string of the form
"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard text @ a particular logging frequency.txt_fn (Callable, Iterable[Callable]) – Preprocess variable before logging to tensorboard. If
txt
is a list thentxt_fn
can also be passed as list with a callable for each entry intxt
or can be passed as a single callable used for all entries.txt_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
txt_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
vid (str, Iterable[str]) – A modulo string of the form
"{regex}@{steps}s"
(or list of) giving the variables to add as tensorboard videos @ a particular logging frequency.vid_fn (Callable, Iterable[Callable]) – Preprocess variable to a tensor of shape [B, T, C, H, W] before logging to tensorboard. If
vid
is a list thenvid_fn
can also be passed as list with a callable for each entry invid
or can be passed as a single callable used for all entries.vid_pref (str, Iterable[str]) – Prefix all summaries in tensorboard with this string
vid_prep (bool, Iterable[bool]) – If True then dictionary variables will be prepended by the name of the dictionary
msg (str, Iterable[str]) – A modulo string of the form
"{regex}->{exp_regex}@{steps}s"
(or list of) giving the variables to log as messages with the experiments matchingexp_regex
.msg_keep (str, Iterable[str]) – One of [‘latest’, ‘max’, ‘min’] giving the protocol to use on message collision
msg_leader (str, Iterable[str]) – A regex to a single variable. If supplied then this variable will be treated as the leader and all other variables will be logged only if the keep condition is met for the leader
msg_expand (bool, Iterable[bool]) – If True then dictionary variables with K keys will be expanded to give K variables
msg_prep (bool, Iterable[bool]) – If True then each variable in the dictionary will be prepended by the dictionary name.
probe (str, Iterable[str]) – A string of the form
f"{regex}@{steps}"
to log resource use to each experiment that matches regex (or list of for different triggers).limit (str) – A modulo string of the form ``f”@{modulo}{triger}” used to limit the number of iterations of an experiment at a particular trigger level. This is useful if an experiment is restarted for example.
Note
All variables …_fn, …_pref and …_prep as well as
ckpt_keep
andmsg_leader
,msg_expand
andmsg_prep
can be supplied as a single or as a list of entries, one for each set of variables matching each modulo string in each case.Example 1:
nn, opt, dataset = ..., ... m = Monitor( directory, checkpoint=('model@1e', 'opt@100s'), # Checkpoint the model once per epoch and opt every 100 steps log='^loss$@100s', # Log the loss to stdout every 100 steps img='^x$@1000s', sca=('^loss$@100s', 'eval_.*@1e'), time=('@100s') # Log to tensorboard time=('@100s', ), # Generate timing statistics every 100 steps hooks=[ # Custom hooks are also supported MyVeryOwnHook(...)]) # The only modification needed to the training loop are the em calls. # Nested loops corresponds to different triggers from inside out # we have ["step" or "s", "epoch" or "e", "era" or "er", "eon" or "eo", "supereon" or "se"] for epoch in m(range(10)): for x, y in m(datset): _y_ = model(x) opt.zero_grad() loss = loss_fn(y, _y_) loss.backward() loss.step() em.log('Manual Logging is also supported') eval_1, eval_2 = eval(model, ds) # Steps and epoch have been incremented assert em.step == len(ds) * 10 assert em.epoch == 10 # Lets reload the model at the 5th epoch em.load(step=5*len(ds), model) # The step and epoch will be updated print(em.step, em.epoch)
Example 2:
from xmen.monitor import Monitor import numpy as np import torch import os import matplotlib.pyplot as plt from torchvision.datasets.mnist import MNIST from torch.utils.data import DataLoader import torchvision.transforms as T plt.style.use('ggplot') ds = DataLoader(MNIST(os.getenv("HOME") + '/data/mnist', download=True, transform=T.Compose( [T.Resize([64, 64]), T.CenterCrop([64, 64]), T.ToTensor(), T.Normalize([0.5], [0.5])])), 8) m = Monitor( directory='/tmp/tb_5', sca=['^z$|^X$@10s', '^a|^x$@5s'], img=['^mnist@10s', '^mnist@5s'], img_fn=[lambda x: x[:2], lambda x: x[:5]], img_pref=['2', '5'], hist='Z@1s', fig='fig@10s', txt='i@5s', txt_fn=lambda x: f'Hello at step {x}', vid='^mnist@10s', vid_fn=lambda x: (x.unsqueeze(0) - x.min()) / (x.max() - x.min()) ) # variables x = 0. a = [1, 2] z = {'x': 5, 'y': 10} for i, (mnist, _) in m(zip(range(31), ds)): # plot a figure fig = plt.figure(figsize=[10, 5]) plt.plot(np.linspace(0, 1000), np.cos(np.linspace(0, 1000) * i)) # random tensor Z = torch.randn([10, 3, 64, 64]) * i / 100 # scalars x = (i - 15) ** 2 z['i'] = i z['x'] += 1 z['y'] = z['x'] ** 2
-
checkpoint
(**kwargs)[source]¶ Checkpoint the torch.nn objects with step and epoch passed as
name==variable_to_save
-
load
(directory=None, step=None, attempt=True, update_triggers=True, **modules)[source]¶ Load the torch torch.nn objects passed as name=variable_to_load, from the directory and reset the state of the em (if update_triggers == True). If attempt == False then an Exception will be raised if either the directory does not contain checkpoints corresponding to modules.
-
Hooks¶
-
class
xmen.monitor.
Hook
(spec)[source]¶ A base class defining a variable passing protocol with the Monitor class. Ever time the monitors count is divisible by
modulo
for a particulartrigger
then the hook is passed all the variables matchingregex
from the current stack. Users therefore define hooks by overloading the hooks __call__ method (with the same call signature).
-
class
xmen.monitor.
XmenMessenger
(spec, keep='latest', leader=None, expand=False, prepend=None)[source]¶ A hook for logging messages with an
xmen.Experiment
object.Example 1:
from xmen import Experiment from xmen.monitor import Monitor messenger = XmenMessenger('y.*->ex.*@10s') # log all variables matching the loss to experiments matching ex m = Monitor(hooks=[messenger]) y1, y2 = 0, 0 ex1, ex2 = Experiment(), Experiment() ex1.link('/tmp', 'ex1') ex2.link('/tmp', 'ex2') for i in m(range(40)): y1 += 1 y2 += 2 if i % 10 == 1: print(', '.join( [f"ex1: {k} = {ex1.messages.get(k, None)}" for k in ('y1', 'y2')] + [f"ex2: {k} = {ex1.messages.get(k, None)}" for k in ('y1', 'y2')])) # Output # ex1: y1 = None, ex1: y2 = None, ex2: y1 = None, ex2: y2 = None # ex1: y1 = 10, ex1: y2 = 20, ex2: y1 = 10, ex2: y2 = 20 # ex1: y1 = 20, ex1: y2 = 40, ex2: y1 = 20, ex2: y2 = 40 # ex1: y1 = 30, ex1: y2 = 60, ex2: y1 = 30, ex2: y2 = 60
Example 2:
from xmen import Experiment from xmen.monitor import Monitor ex = Experiment() ex.link('/tmp', 'ex') m = Monitor( hooks=[ XmenMessenger('^y$|^i$->^ex$@10s', keep='min', leader='^y$')]) x = -50 for i in m(range(100)): x += 1 y = x ** 2 if i % 10 == 1: print([ex.messages.get(k, None) for k in ('i', 'y')]) # Output # [None, None] # [9, 1600] # [19, 900] # [29, 400] # [39, 100] # [49, 0] # [49, 0] # [49, 0] # [49, 0] # [49, 0]
Example 3:
from xmen import Experiment from xmen.monitor import Monitor ex = Experiment() ex.link('/tmp', 'ex') m = Monitor( hooks=[ XmenMessenger('z->^ex$@10s', keep='min', leader='y', expand=True)]) z = {'x': 5, 'y': 10} for i in m(range(100)): z['i'] = i z['x'] += 1 z['y'] = z['x'] ** 2 if i % 10 == 1: print([ex.messages.get(k, None) for k in ('i', 'y', 'x')]) # [None, None, None] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15] # [9, 225, 15]
Example 4:
from xmen import Experiment from xmen.monitor import Monitor ex = Experiment() ex.link('/tmp', 'ex') m = Monitor( hooks=[ XmenMessenger('z->^ex$@10s', keep='min', leader='z_y', expand=True, prepend=True)]) z = {'x': 5, 'y': 10} for i in m(range(100)): z['i'] = i z['x'] += 1 z['y'] = z['x'] ** 2 if i % 10 == 1: print([ex.messages.get(k, None) for k in ('z_i', 'z_y', 'z_x')]) # same output as above
-
__init__
(spec, keep='latest', leader=None, expand=False, prepend=None)[source]¶ - Parameters
spec (Spec) – hook specification in the form either
{log_regex}->{exp_regex}@{modulo}{trigger}
or{exp_regex}@{modulo}{trigger}
whereexp_regex
is the experiments to message andlog_regex
are additional messages to log with the experiment. In the second case only timing and step information will be loggedkeep (str) – One of [‘latest’, ‘max’, ‘min’] in the case of message collision with each experiment
leader (regex) – If leader is not None then messages will be logged according to
keep
as judged by the variable in var_dict which matchesleader
.expand – If a dictionary variable with K keys matches
log_regex
then it is converted to K variables with names corresponding to the keys in dict ifexpand==True
.prepend – If prepend is
True
then in the case above the name of each variable will be prepended by the dict variable name. eg. each variable will be called ‘{name}_{k}’.
-
-
class
xmen.monitor.
Timer
(spec)[source]¶ A simple timing hook used to log any timers open_socket by the experiment monitor
-
__init__
(spec)¶ Initialize self. See help(type(self)) for accurate signature.
-
-
class
xmen.monitor.
Logger
(spec, format='', process_func=None)[source]¶ A simple logging hook used to log variables to stdout.
Example:
from xmen.monitor import Monitor, Logger m = Monitor( hooks=[ Logger('x@2s', process_func=lambda x: '|'.join(x)), Logger('y@1e', format='.5f')]) x = ['cat', 'dog'] y = 0. for _ in m(range(3)): for i in m(range(5)): y += i # [01:17PM 18/11/20 0/3 2/15]: x = cat|do # [01:17PM 18/11/20 0/3 4/15]: x = cat|do # [01:17PM 18/11/20 1/3]: y = 10.0000 # [01:17PM 18/11/20 1/3 6/15]: x = cat|do # [01:17PM 18/11/20 1/3 8/15]: x = cat|do # [01:17PM 18/11/20 1/3 10/15]: x = cat|do # [01:17PM 18/11/20 2/3]: y = 20.0000 # [01:17PM 18/11/20 2/3 12/15]: x = cat|do # [01:17PM 18/11/20 2/3 14/15]: x = cat|do # [01:17PM 18/11/20 3/3]: y = 30.0000
-
__init__
(spec, format='', process_func=None)[source]¶ - Parameters
spec (Spec) – a specification string of the form “{regex}@{modulo}{trigger}”. Variables matching
regex
will be logged whentrigger
%modulo
== 0.format (str) – a format string used as f”{var:format}” for logging string variables
process_func (callable) – used to convert variables to a string for logging to stdout. Format will be applied after.
-
-
class
xmen.monitor.
TensorboardLogger
(type, spec, fn=None, prefix='', prepend=True, **options)[source]¶ A hook for logging tensorboard summaries. Currently
image
,scalar
,histogram
,figure
,video
,text
,pr_curve
,mesh
are all supported.Before being passed to the summary writer each variable is processes as follows:
First all variables are passed to
fn
(if supplied).Variables of type list or dict with length K will be expanded to give K variables. The name of each variable will be the list name postpended with its index or the dictionary name postpended with its key.
If the summary type is image or scalar then some additional processing will be performed:
For scalars if variables are not already scalar variables they will be converted by calling var.mean()
For images the variable must be either a 3D [C, H, W] or 4D [B, C, H, W] tensor. Tensors are converted to images of shape [C, H, W]:
if the variable is 3D it will be converted to 4D
the variable is then converted to an image using
torchvision.utils.make_grid
. Additional options can be passed totorchvision.utils.make_grid
using theoptions
parameter. Available options include:'nrow' # images per row
'padding'
'normalize'
'range'
'scale_each'
'pad_value'
(see torchvision.utils.make_grid for more info)
Note
Tensorboard does not allow summaries to have the same name. If you want to leave to different types of summary for the same variable then you will need to use the prefix argument.
-
__init__
(type, spec, fn=None, prefix='', prepend=True, **options)[source]¶ - Parameters
type (str) – The type of tensorboard summary to log. Should be one of [‘image’, ‘scalar’, ‘histogram’, ‘figure’, ‘video’, ‘text’, ‘pr_curve’, ‘mesh’]
spec (str) – A string in the form
"{regex}@{modulo}{trigger}"
. tensorbaord summaries will be logged for all variables matchingregex
when ``monitor.{trigger} % modulo == 0`fn (callable) – A function used to convert each variable to a summary
prefix (str) – prepend all summaries with this string
prepend (bool) – If true prepend dictionary variable names with the dictionary name.
**options – Keyword arguments passed to
torchvision.utils.make_grid
Helpers¶
-
xmen.monitor.
load
(load_dir, step=None, attempt=True, **modules)[source]¶ Load torch objects from a checkpoint object. If
Step is None
then the most recent checkpoint is loaded. Else the checkpoint is loaded at the specified step. If attempt == False then load will raise a ValueError if either one of the modules was not found in modules or if no checkpoints were found in load_dir.Note
No check to ensure the meta information in each file is the same. The meta information returned corresponds to the first module encountered in modules.
lightning¶
utils¶
A module containing several utilitity functions, classes and Meta Classes used by the ExperimentManager and the Experiment classes.
-
xmen.utils.
save_param
(params, root, file='params.yml')[source]¶ Save a dictionary of parameters at
{root}/params.yml
- Parameters
params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel.yaml
root (str) – The root of the experiment
-
xmen.utils.
load_params
(roots)[source]¶ Load params.yml files into a list of dictionaries from a list of paths
-
xmen.utils.
flatten
(d, parent_key='', sep='_')[source]¶ Flatten a nested dictionary to a single dictionary. The keys of nested entries will be joined using sep
-
xmen.utils.
dic_to_yaml
(dic, typ='rt', default_flow_style=False)[source]¶ Convert dictionary to a yaml string (
dic
can also be a CommentedMap)
-
xmen.utils.
dic_from_yml
(*, string=None, path=None)[source]¶ Load from either a yaml string or path to a yaml file
-
xmen.utils.
get_size
(bytes, suffix='B')[source]¶ Scale bytes to its proper format e.g:
1253656 => ‘1.20MB’ 1253656678 => ‘1.17GB’
-
xmen.utils.
get_meta
(get_platform=False, get_cpu=False, get_memory=False, get_disk=False, get_slurm=False, get_conda=False, get_network=False, get_gpu=False, get_environ=False, live=False)[source]¶ Get Meta information for the system
-
xmen.utils.
get_attribute_helps
(cls)[source]¶ Get all help files for class
cls
from doc strings from all inherited classes.
-
class
xmen.utils.
TypedMeta
(name, bases, attr_dict)[source]¶ A meta class helper used to generate automatic doc strings generated from typed class definitions. This allows the docstrings to be defined inside the
__init__
body instead of inside the doc string iradicating the need to type attributes twice:MyClass(metaclass=TypedMeta): '''This class is bound to do something...''' a int: 3 # @p My first attribute b int: 5 # @p My second attribute m = MyClass() print(m.__init.__.__doc__)
Now if we call:
>>> ob = print(MyClass.__doc__) This class is bound to do something... Parameters: a (int) : My first attribute (default=3) b (int) : My second attribute (default=5)
The doc string has automatically been updated.
-
xmen.utils.
get_git
(path)[source]¶ Get git information for the given path.
- Returns
- Empty if status == False otherwise with keys:
remote: A url to the remote repository
hash: A git hash to the current commit
- Return type
(dict)
- status (bool): False if either git is not available, the repo is not in a git repository or if there are
uncommited changes in the current repository. Otherwise true
-
xmen.utils.
get_version
(*, path=None, cls=None, fn=None)[source]¶ Get version information for either a path to a directory or a class. Git version information is loaded if available.
- Parameters
path (str) – A path to a repository which is inspected for version information
cls (Class) – A Class object to be inspected for version information
- Returns
- A dictionary containing at least one of the following:
if path is not None:
'path'
: A copy of the path
if cls is not None:
'module'
: A path to the module in which the class was defined'class'
: The name of the class
if git != {}
: (i.e if path or module is in a git repository):'git_local'
: The root of the local git repository'git_commit'
: The hash of the commit'git_remote'
: The remote url if has remote elseNone
- Return type
version (dict)
-
xmen.utils.
get_run_script
(module, name, shell='/usr/bin/env python3', comment='#')[source]¶ Generate a run script for a particular experiment.
- Parameters
module (str) – the module to look in
name (str) – the name of the experiment in the module. If name corresponds to a function it will be converted to an Experiment class
manager¶
-
class
xmen.manager.
ExperimentManager
(root='', headless=False)[source]¶ A helper class with wrapped command line interface used to manage a set of experiments. It is compatible both with experiments generated by the
Experiment.to_root
method call as well as any experiment that can be represented as a bash script.sh which takes a set of parameters in a yaml file as input.- More Info:
At its core the experiment manager maintains a single root directory:
root ├── defaults.yml ├── experiment.yml ├── script.sh ├── {param}:{value}__{param}:{value} │ └── params.yml ├── {param}:{value}__{param}:{value} │ └── params.yml ...
In the above we have:
defaults.yml
defines a set of parameter keys and there default values shared by each experiment. It generated from theExperiment
class it will look like:# Optional additional Meta parameters _created: 06:58PM September 16, 2019 # The date the defaults were created _version: module: /path/to/module/experiment/that/generated/defaults/was/defined/in class: TheNameOfTheExperiment git: local: /path/to/git/repo/defaults/were/defined/in branch: some_branch remote: /remote/repo/url commit: 80dcfd98e6c3c17e1bafa72ee56744d4a6e30e80 # The git commit defaults were generatd at # Default parameters a: 3 # This is the first parameter (default=3) b: '4' # This is the second parameter (default='4') The ``ExperimentManager`` is also compatible with generic experiments. In this the ``_version`` meta field can be added manually, replacing ``module`` and ``class`` with ``path``. The git information for each will be updated automatically provided ``path`` is within a git repository.
script.sh
is a bash script. When run it takes a single argument'params.yml'
(eg.`script.sh params.yml`
).Note
Experiment
objects are able to automatically generate script.sh files that look like this:#!/bin/bash # File generated on the 06:34PM September 13, 2019 # GIT: # - repo /path/to/project/module/ # - remote {/path/to/project/module/url} # - commit 51ad1eae73a2082e7636056fcd06e112d3fbca9c export PYTHONPATH="${PYTHONPATH}:path/to/project" experiments /path/to/project/module/experiment.py --execute ${1}
Generic experiments are compatible with the
ExperimentManager
provided they can be executed with a shell script. For example a bash only experiment might have ascript.sh
script that looks like:#!/bin/bash echo "$(cat ${1})"
A set of experiment folders representing individual experiments within which each experiment has a
params.yml
with a set of override parameters changed from the original defaults. These overrides define the unique name of each experiment (in the case that multiple experiments share the same overrides each experiment folder is additionally numbered after the first instantiation). Additionally, eachparams.yml
contains the following:# Parameters special to params.yml _root: /path/to/root # The root directory to which the experiment belongs (should not be set) _name: a:10__b:3 # The name of the experiment (should not be set) _status: registered # The status of the experiment (one of ['registered' | 'running' | 'error' | 'finished']) _created: 07:41PM September 16, 2019 # The date the experiment was created (should not be set) _purpose: this is an experiment example # The purpose for the experiment (should not be set) _messages: {} # A dictionary of messages which are able to vary throughout the experiment (should not be set) # git information is updated at registration if ``_version['module']`` or `_version['path']`` # exists in the defaults.yml file and the path is to a valid git repo. _version: module: /path/to/module # Path to module where experiment was generated class: NameOfExperimentClass # Name of experiment class params are compatible with git: # A dictionary containing the git history corresponding to the defaults.yml file. Only local_path: path/to/git/repo/params/were/defined/in remote_url: /remote/repo/url hash: 80dcfd98e6c3c17e1bafa72ee56744d4a6e30e80 # Parameters from the default (with values overridden) a: 3 # This is the first parameter (default=3) b: '4' # This is the second parameter (default='4')
experiment.yml
preserves the experiment state with the following entries:root: /path/to/root defaults: /path/to/root/defaults.yml script: /path/to/root/script.sh experiments: - /private/tmp/new-test/a:10__b:3 - /private/tmp/new-test/a:20__b:3 - /private/tmp/new-test/a:10__b:3_1 overides: - a: 10 b: '3' - a: 20 b: '3' - a: 10 b: '3' created: 07:41PM September 16, 2019 # The date the experiment manager was initialised
The
ExperimentManager
provides the following public interface for managing experiments:__init__(root)
:Link the experiment manager with a root directory and load the experiments.yml if it exists
initialise(script, defaults)
:Initialise an experiment set with a given script and default parameters
link(string_pattern)
:Register a number of experiments overriding parameters based on the particular
string_pattern
list()
:Print all the experiments and their associated information
unlink(pattern)
:Relieve the experiment manager of responsibility for all experiment names matching pattern
clean()
:Delete any experiments which are no longer the responsibility of the experiment manager
run(string, options)
:Run an experiment or all experiments (if string is
'all'
) with options prepended.
Example:
experiment_manager = ExperimentManager(ROOT_PATH) # Create experiment set in ROOT_PATH experiment_manager.initialise(PATH_TO_SCRIPT, PATH_TO_DEFAULTS) experiment_manager.link('parama: 1, paramb: [x, y]') # Register a set of experiments experiment_manger.unlink('parama_1__paramb_y') # Remove an experiment experiment_manager.clean() # Get rid of any experiments no longer managed experiment_run('parama:1__paramb:x', sh) # Run an experiment experiment_run('all') # Run all created experiments
-
__init__
(root='', headless=False)[source]¶ Link an experiment manager to root. If root already contains an
experiment.yml
then it is loaded.In order to link a new experiment with a defaults.yml and script.sh file then the initialise method must be called.
- Parameters
root – The root directory within which to create the experiment. If “” then the current working directory is used. If the root directory does not exist it will be made.
root – The root directory of the experiment manger
defaults – A path to the defaults.yml file. Will be None for a fresh experiment manager (if experiments.yml has just been created).
script – A path to the script.sh file. Will be None for a fresh experiment manager (if experiments.yml has just been created).
created – A string giving the date-time the experiment was created
experiments – A list of paths to the experiments managed by the experiment manager
overides – A list of dictionaries giving the names (keys) and values of the parameters overridden from the defaults for each experiment in experiments.
notes – A set of notes attched to the experiment set
purpose – The purpose of the epxeriment
-
check_initialised
()[source]¶ Make sure that
'experiment.yml'
,'script.sh'
,'defaults.yml'
all exist in the directory
-
save_params
(params, root)[source]¶ Save a dictionary of parameters at
{root}/{experiment_name}/params.yml
- Parameters
params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel
experiment_name (str) – The name of the experiment
-
update_meta
()[source]¶ Save a dictionary of parameters at
{root}/{experiment_name}/params.yml
- Parameters
params (dict) – A dictionary of parameters to be saved. Can also be a CommentedMap from ruamel
experiment_name (str) – The name of the experiment
-
load_params
(experiment_path, experiment_name=False)[source]¶ Load parameters for an experiment. If
experiment_name
is True then experiment_path is assumed to be a path to the folder of the experiment else it is assumed to be a path to theparams.yml
file.
-
initialise
(*, defaults='', script='', purpose='', name=None)[source]¶ Link an experiment manager with a
defaults.yml
file andsript.sh
.- Parameters
defaults (str) – A path to a
defaults.yml
. If “” then adefaults.yml
is searched for in the current work directory.script (str) – A path to a
script.sh
. If""
then a script.sh file is searched for in the current work directory.
-
note
(pattern, msg, remove=False)[source]¶ Add a note to experiments matching pattern. If remove is True msg is deleted instead.
-
register
(name=None, string_params=None, purpose='', header=None, shell='/bin/bash', repeats=1)[source]¶ Register a set of experiments with the experiment manager.
Experiments are created by passing a yaml dictionary string of parameters to overload in the
params.yml
file. The special symbol'|'
can be thought of as an or operator. When encountered each of the parameters either side'|'
will be created separately with all other parameter combinations.- Parameters
string_params (str) –
A yaml dictionary of parameters to override of the form
'{p1: val11 | val12, p2: val2, p3: val2 | p4: val31 | val32 | val33, ...}'
. The type of each parameter is inferred from its value in defaults. A ValueError will be raised if any of the parameter cannot be found in defaults. Parameters can be float (1.), int (1), str (a), None, and dictionaries {a: 1., 2.} or lists [1, 2] of these types_match. None parameters are specified using empty space. The length of list parameters must match the length of the parameter in default. Dictionary parameters may only be partially defined. Missing keys will be assumed to take there default value.The special character ‘|’ is used as an or operator. All combinations of parameters either side of an | operator will be created as separate experiments. In the example above
N = 2 * 2 * 3 = 12
experiments will be generated representing all the possible values for parametersp1
,p3
andp4
can take withp2
set toval2
for all.purpose (str) – An optional purpose message for the experiment.
header (str) – An optional header message prepended to each run script.sh
Note
This function is currently only able to link list or dictionary parameters at the first level.
{a: {a: 1.. b: 2.} | {a: 2.. b: 2.}}
works creating two experiments with over-ridden dicts in each case but{a: {a: 1. | 2., b:2.}}
will fail.The type of each override is inferred from the type contained in the defaults.yml file (ints will be cast to floats etc.) where possible. This is not the case when there is an optional parameter that can take a None value. If None (null yaml) is passed as a value it will not be cast. If a default.yml entry is given the value null the type of any overrides in this case will be inferred from the yaml string.
-
reset
(pattern, status='registered')[source]¶ Update the status of the experiment. This is useful if you need to re-run an experiment from a latest saved checkpoint for example.
- Parameters
pattern – The experiment name
-
run
(pattern, *flags)[source]¶ Run all experiments that match the global pattern using the run command given by args.
config¶
-
class
xmen.config.
GlobalExperimentManager
[source]¶ DEPRECIATED A helper class used to manage global configuration of the Experiment Manager
-
clean
()[source]¶ Iteratively search through experiment sets and remove any that no longer exist.
Note
This is performed by checking that experiment folder is a valid set using using
ExperimentManager(p).check_initialised()
-
find
(mode='all', last=None, pattern='*', param_match=None, types_match=None, load_defaults=False, missing_entry='')[source]¶ Search through all experiments in the globally filtering by experiment location, parameters, and type.
- Parameters
mode (str) – If ‘all’ then all experiments for each set will be concatenated and summarised individually whilst if ‘set’ then experiments will be summarised and searched for based on the default parameters.
paths (List[str]) – Extract all information from the passed list of path to experiments. If None then a search will be performed instead.
pattern (str) – A unix glob pattern of experiments to consider
param_match (list) – A list of strings providing a condition to filter experiments by their parameters. Compatible string formats are “regex”, “regex op val” or “val1 op1 regex op2 val2” where op is in
[<=,==,>=,<,>,!=]
. If just a regex is supplied then only experiments with parameters matching the regex will be returned. If an op is supplied then only experiments with all parameters which match the regex and satisfy the condition will be returned.types_match (list) – A list of types of experiments to search for
load_defaults (bool) – If True then
defaults.yml
files will be loaded and the parameters of each experiment inferred from the overides defined in theexperiment.yml
file for each experiment set. This is potentially faster (less io) but means that the messages for each experiment will not be returned.missing_entry – Assign this parameter to missing values (see notes).
- Returns
- A dict with possible keys ``[‘_root’, ‘_purpose’, ‘_type’, ‘_notes’, ‘_name’, ‘_created’,
’_version’, ‘_status’, ‘_messages’, ‘_experiments’, *param_keys]`` where
param_keys
correspond to any parameters that satisfy the conditions in param_match. The keys['_root', '_purpose', '_type', '_notes', '_created']
will always be present,['_name', '_version']
will be added ifload_defaults==True
and['_name', '_version', '_status', '_messages]
ifload_defaults==False
. If [mode == ‘set’] then the key ‘_experiments’ will also be added (giving a list of paths to the experiments in each set). The created dates in this case correspond to the date the set was created (given in defaults.yml).
special_keys (dict): A list of keys starting with ‘_’ present in the dictionary.
- Return type
matches (dict)
Note
- Due to the use of regex and conditions it is possible for parameters to match in one experiment which are
not present in another. In this case missing parameters are given a value equal to
missing_string
.
-
find_to_dataframe
(table, verbose=True, display_git=None, display_purpose=None, display_date=None, display_messages=None, display_status=None, display_meta=None, display_params=None)[source]¶ Convert the output of the find method to a formatted data frame configured by args. If verbose then all entries will be displayed (independent of the other args)
-