API
Nodes
Suite
- class pyflow.Suite(name, host=None, exit_hook=None, *args, **kwargs)
- __init__(name, host=None, exit_hook=None, *args, **kwargs)
Represents a collection of interrelated ecFlow tasks.
- Parameters:
name (str) – Name of the suite to create.
host (Host) – The host to execute the suite on. If None, default ecFlow behaviour will be used.
exit_hook (str,list) – a script containing some commands to be called at exit time.
json (dict) – Parsed JSON for creation of the children node(s).
workdir (str) – The working directory for the tasks, can be fixed or an ecFlow variable.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the suite is a shadow node created to satisfy an Extern, and should not be generated.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
generated_variables (GeneratedVariable) – An attribute for setting an ecFlow generated variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
repeat (Repeat) – An attribute for setting a repeat schedule for the node.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the suite.
Example:
with pyflow.Suite('s', host=pyflow.LocalHost(), defstatus=pyflow.state.suspended, FOO='BAR') as s: pass
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- deploy_suite(target=<class 'pyflow.deployment.FileSystem'>, node=None, **options)
Deploys suite and its components.
- Parameters:
target (Deployment) – Deployment target for the suite.
node (str) – Path to node to limit deployment to a family/task.
**options (dict) – Accept extra keyword arguments as deployment options.
- Returns:
Deployment target object.
- Return type:
Deployment
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow suite object.
- Returns:
ecFlow suite object.
- Return type:
ecflow.Suite
- find_node(subpath)
Returns node under provided relative path.
- Parameters:
subpath (str) – Relative path of the node to search for.
- Returns:
Found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- relative_path(node)
Returns relative path of the suite.
- Parameters:
node (Node) – Unused.
- Returns:
Relative path of the suite.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of modules.
- Returns:
List of modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The anchor object.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property files_path
The files path of the node.
- Type:
str
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property include_path
The include path of the node.
- Type:
str
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
Task
- class pyflow.Task(name, autolimit=True, submit_arguments=None, exit_hook=None, clean_workdir=False, **kwargs)
- __init__(name, autolimit=True, submit_arguments=None, exit_hook=None, clean_workdir=False, **kwargs)
Describes what should be carried out as one executable unit within an ecFlow suite.
- Parameters:
autolimit (bool) – Whether to automatically add the task to the executing hosts limit, if it has one.
submit_arguments (dict, str) – Job submission arguments, can be passed as a dictionary or a string pointing to an entry in the dictionary given to the host.
exit_hook (str,list) – a script containing some commands to be called at exit time.
clean_workdir (bool) – Whether to ensure that the working directory is empty.+
script (str,list) – The script command or the list of script commands associated with the task.
json (dict) – Parsed JSON for creation of the children node(s).
host (Host) – The host to execute the task on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the task is a shadow node created to satisfy an Extern, and should not be generated.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
mirror (Mirror) – An attribute for a mirroring a node on another ecflow server.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
generated_variables (GeneratedVariable) – An attribute for setting an ecFlow generated variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
repeat (Repeat) – An attribute for setting a repeat schedule for the node.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the task.
Example:
with pyflow.Task('t', script='echo "Hello, world!"', FOO='bar') as t: pass
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow task object.
- Returns:
ecFlow task object.
- Return type:
ecflow.Task
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_script()
Generates the complete script for the task.
- Returns:
Complete script for the task.
- Return type:
str
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of task modules.
- Returns:
List of task modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the host or task modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property deploy_extension
The script file extension to be used during deployment of the task.
- Type:
str
- property deploy_path
The deployment path of the current task, may be None.
- Type:
str
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submit_arguments
The dictionary of submit arguments.
- Type:
dict
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
Family
- class pyflow.Family(name, json=None, modules=None, purge_modules=False, extern=False, exit_hook=None, **kwargs)
- __init__(name, json=None, modules=None, purge_modules=False, extern=False, exit_hook=None, **kwargs)
Provides both visual and logical grouping of related families and tasks.
- Parameters:
name (str) – The name of the family to create.
json (dict) – Parsed JSON for creating the children node(s).
host (Host) – The host to execute the family on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the family is a shadow node created to satisfy an Extern, and should not be generated.
exit_hook (str,list) – a script containing some commands to be called at exit time.
workdir (string) – Working directory for tasks.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
generated_variables (GeneratedVariable) – An attribute for setting an ecFlow generated variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
repeat (Repeat) – An attribute for setting a repeat schedule for the node.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the family.
Example:
with pyflow.Family('f', labels={'foo': 'bar'}) as f: pass
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow family object.
- Returns:
ecFlow family object.
- Return type:
ecflow.Family
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of modules.
- Returns:
List of modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property manual_path
The deployment path of the current task, may be None.
- Type:
str
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
AnchorFamily
- class pyflow.AnchorFamily(name, json=None, modules=None, purge_modules=False, extern=False, exit_hook=None, **kwargs)
- __init__(name, json=None, modules=None, purge_modules=False, extern=False, exit_hook=None, **kwargs)
Provides grouping of tasks that require encapsulation.
- Parameters:
name (str) – Name of the anchor family to create.
json (dict) – Parsed JSON for creation of the children node(s).
host (Host) – The host to execute the anchor family on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the anchor family is a shadow node created to satisfy an Extern, and should not be generated.
exit_hook (str,list) – a script containing some commands to be called at exit time.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
repeat (Repeat) – An attribute for setting a repeat schedule for the node.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the anchor family.
Example:
with pyflow.AnchorFamily('af', labels={'foo': 'bar'}) as af: pass
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow family object.
- Returns:
ecFlow family object.
- Return type:
ecflow.Family
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of modules.
- Returns:
List of modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The anchor object.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property files_path
The files path of the node.
- Type:
str
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property include_path
The include path of the node.
- Type:
str
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property manual_path
The deployment path of the current task, may be None.
- Type:
str
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
Attributes
Variable
- class pyflow.Variable(name, value)
An attribute for setting an ecFlow variable.
- Parameters:
name (str) – The name of the variable.
value (str) – The value of the variable.
Example:
Variable('FOO', 'foo_value')
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.Edit(**kwargs)
An attribute for setting multiple ecFlow variables.
- Parameters:
**kwargs (dict) – Accept keyword arguments as variables to be set.
Example:
pyflow.Edit(FOO='foo_value', BAR='bar_value')
GeneratedVariable
- class pyflow.GeneratedVariable(name)
An attribute for referencing an ecFlow generated variable. The variable value will be generated automatically by ecFlow.
- Parameters:
name (str) – The name of the variable.
Example:
GeneratedVariable('FOO')
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Trigger
- class pyflow.Trigger(value, *args)
An attribute for setting a condition for running the node depending on other tasks or families.
- Parameters:
value (expression) – Expression to evaluate for running the node.
*args (tuple) – Accept extra positional arguments for expressions provided as a JSON value.
Example:
pyflow.Trigger(t1 & t2)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Event
- class pyflow.Event(name)
An attribute for declaring an action that a node can trigger while it is running.
- Parameters:
name (str) – The name of the event.
Example:
pyflow.Event('a')
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Complete
- class pyflow.Complete(value)
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
Note
Complete expression evaluation takes precedence over the trigger.
- Parameters:
value (expression) – Expression to evaluate for setting the node as complete.
Example:
pyflow.Complete(t1 & t2)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Late
- class pyflow.Late(value)
An attribute for a flag for notifying if task does not run as expected.
- Parameters:
value (str) – Expression to evaluate for setting the late flag.
Example:
pyflow.Late('-c +00:01') # set late flag if task takes longer than a minute
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Label
- class pyflow.attributes.Label(name, value)
An attribute for a string value that can be set from a script.
- Parameters:
name (str) – The name of the label.
value (str) – The initial value of the label.
Example:
pyflow.attributes.Label('foo', 'bar')
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Meter
- class pyflow.Meter(name, min, max=None, threshold=None)
An attribute for a range of integer values that can be set from a script.
- Parameters:
name (str) – The name of the meter.
min (int,tuple,list) – Minimum value of the meter. Alternatively, a tuple or list containing minimum, maximum and threshold value of the meter.
max (int) – Maximum value of the meter.
threshold (int) – Threshold value of the meter.
Example:
pyflow.Meter('progress', 1, 100, 90)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Defstatus
- class pyflow.Defstatus(value)
An attribute for declaring the default status of the node.
- Parameters:
value (State) – Default state to set for the node.
Example:
pyflow.Defstatus(pyflow.state.suspended)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Aviso
- class pyflow.Aviso(name, listener, url, schema, polling, auth)
An attribute that allows a node to be triggered by an external Aviso notification.
- Parameters:
name (str) – The name of the attribute.
listener (str) – The listener configuration.
url (str) – The URL to the Aviso server.
schema (str) – The schema used to process Aviso notifications.
polling (str, int) – The time interval used to poll the Aviso server.
auth (str) – The path to the Aviso authentication credentials file.
Example:
pyflow.Aviso("AVISO_NOTIFICATION", r'{ "event": "mars", "request": { "class": "od"} }', "https://aviso.ecm:8888/v1", "/path/to/schema.json" 60, "/path/to/auth.json")
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Mirror
- class pyflow.Mirror(name: str, remote_path: str, remote_host: str, remote_port: str, auth: str = '', polling: int = 300, ssl: bool = False, force: bool = False)
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Parameters:
name (str) – The name of the attribute.
remote_path (str) – The path to the mirrored node on the remote ecFlow server.
remote_host (str) – The host used to connect to the remote ecFlow server.
remote_port (str, int) – The port used to connect to the remote ecFlow server.
polling (str, int) – The time interval used to poll the remote ecFlow server.
ssl (bool) – The flag indicating if SSL communication is enabled.
auth (str) – The path to the ecFlow authentication credentials file.
force (bool) – The flag indicating whether to override lower limit on polling time
Example:
pyflow.Mirror("NODE_MIRROR" "/suite/family/task", "remote-ecflow-server", "3141", 300, False "/path/to/auth.json")
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Time Dependencies
- class pyflow.Time(value)
An attribute for setting a time dependency of the node.
- Parameters:
value (str) – Either a cron-like expression (m h d M D), cron-like time series expression (start(hh:mm) end (hh:mm) increment(hh:mm)) or an absolute or relative time stamp (hh:mm).
Example:
pyflow.Time("23:00") # at next 23:00 pyflow.Time("0 10-20 * * *") # every hour between 10 am and 8 pm
- Raises:
AssertionError – Cron-like time definition do not support Day of Week, Day of Month or Month.
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.Cron(value, days_of_week=None, last_week_days_of_the_month=None, days_of_month=None, last_day_of_the_month=None, months=None)
An attribute for setting a cron dependency of the node.
- Parameters:
value (str) – Either a cron-like expression (m h d M D), cron-like time series expression (start(hh:mm) end (hh:mm) increment(hh:mm)) or an absolute or relative time stamp (hh:mm).
days_of_week (list) – The list of the days of the week when the task should run, with 0 being Sunday and 6 Saturday.
last_week_days_of_the_month (list) – The list of the last days of the week of the month when the task should run, with 0 being Sunday and 6 Saturday.
days_of_month (list) – The list of the days of the month when the task should run.
last_day_of_the_month (bool) – Whether the task should run at the last day of the month.
months (list) – The list of the months when the task should run.
Example:
pyflow.Cron("0 23 * * *") # every day at 11 pm pyflow.Cron("0 8-12 * * *") # every hour between 8 and 12 am pyflow.Cron("0 11 * * SUN,TUE") # every Sunday and Tuesday at 11 am pyflow.Cron("0 2 1,15 * *") # every 1st and 15th of each month at 2 am pyflow.Cron("0 14 1 1 *") # every first of January at 2 pm pyflow.Cron("23:00", last_week_days_of_the_month=[5]) # every *last* Friday of month at 11 pm pyflow.Cron("23:00", days_of_month=[1], last_day_of_the_month=True) # every first and last of month at 11 pm
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.Crons(value)
An attribute for setting a cron time series dependency of the node.
- Parameters:
value (str) – A cron-like time series expression (start(hh:mm) end(hh:mm) increment(hh:mm)) for the node dependency.
Example:
pyflow.Crons("00:00 23:59 00:05") # every 5 minutes during the day
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.Date(value, value2=None, value3=None)
An attribute for setting a date dependency of the node.
Note
All string values support wildcards (*).
- Parameters:
value (str,datetime) – Either the day of the month (d), the complete date (d.M.Y) or the datetime object of the date dependency.
value2 (str) – Month of the date dependency (M).
value3 (str) – Year of the date dependency (Y).
Example:
pyflow.Date("31.12.2012") # the 31st of December 2012 pyflow.Date("01.*.*") # every first of the month pyflow.Date("*.10.*") # every day in October pyflow.Date("1.*.2008") # every first of the month, but only in 2008
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.attributes.Day(value)
An attribute for setting a day of the week dependency of the node.
- Parameters:
value (str) – Day of the week of the dependency.
Example:
pyflow.attributes.Day('monday') # every monday
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.attributes.Today(value)
An attribute for setting a cron dependency of the node for the current day.
- Parameters:
value (str) – A cron-like expression (m h d M D) for the node dependency, limited to current day.
Example:
pyflow.attributes.Today("0 12 * * *") # today at 12 pm
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Repeat
- class pyflow.RepeatDate(name, start, end, increment=1)
An attribute that allows a node to be repeated by a date value.
- Parameters:
name (str) – The name of the repeat attribute.
start (datetime) – The start date of the repeat attribute.
end (datetime) – The end date of the repeat attribute.
increment (int) – The increment used to update the date.
Example:
pyflow.RepeatDate('REPEAT_DATE', datetime.date(year=2019, month=1, day=1), datetime.date(year=2019, month=12, day=31))
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property day
The day of the repeat date.
- Type:
int
- property day_of_week
The day of the week of the repeat date.
- Type:
int
- property fullname
The relative path of the attribute.
- Type:
str
- property julian
The Julian date of the repeat date.
- Type:
int
- property month
The month of the repeat date.
- Type:
int
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- property year
The year of the repeat date.
- Type:
int
- class pyflow.attributes.RepeatDateList(name, value)
An attribute that allows a node to be repeated over a list of dates.
- Parameters:
name (str) – The name of the repeat attribute.
list (tuple) – The list of dates for the repeat attribute.
Example:
pyflow.RepeatDateList("REPEAT_DATELIST", ["20000101", "20000102", "20000103", "d", "e"])
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- property values
The list of date values (as integers).
- Type:
list
- class pyflow.attributes.RepeatDay(value)
An attribute that allows a node to be repeated infinitely.
- Parameters:
value (int) – The repeat step.
Example:
pyflow.attributes.RepeatDay(1)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.RepeatInteger(name, start, end, increment=1)
An attribute that allows a node to be repeated by an integer range.
- Parameters:
name (str) – The name of the repeat attribute.
start (int) – The start integer value of the repeat attribute.
end (datetime) – The end integer value of the repeat attribute.
increment (int) – The step amount used to update the integer.
Example:
pyflow.RepeatInteger("REPEAT_INTEGER", 1, 5, 1)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.RepeatEnumerated(name, value)
An attribute that allows a node to be repeated by an enumerated list.
- Parameters:
name (str) – The name of the repeat attribute.
list (tuple) – The list of enumerations for the repeat attribute.
Example:
pyflow.RepeatEnumerated("REPEAT_STRING", [1, 3, 4, 5])
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- property values
The list of enumerated values.
- Type:
list
- class pyflow.attributes.RepeatString(name, value)
An attribute that allows a node to be repeated by a string value.
- Parameters:
name (str) – The name of the repeat attribute.
list (tuple) – The list of string values for the repeat attribute.
Example:
pyflow.RepeatString("COLOR", ["red", "green", "blue"])
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- property values
The list of repeat string values.
- Type:
list
Limit
- class pyflow.Limit(name, value)
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Parameters:
name (str) – The name of the limit.
value (int) – The maximum number of tasks.
Example:
pyflow.Limit('l', 3)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.InLimit(value)
An attribute for grouping of nodes to which a limit should be applied.
- Parameters:
value (str,Limit_) – The name of the limit or a limit object.
Example:
l = pyflow.Limit('l', 3) pyflow.InLimit(l)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
- class pyflow.Inlimit(value)
An attribute for grouping of tasks to which a limit should be applied.
- Parameters:
value (str,Limit_) – The name of the limit or a limit object.
Example:
l = pyflow.Limit('l', 3) pyflow.Inlimit(l)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Manual
- class pyflow.attributes.Manual(value)
An attribute for setting help text of the node.
- Parameters:
value (str) – The help text or list of help texts to include in the node.
Example:
pyflow.attributes.Manual('This is a multi-line manual\nwhich can contain instructions')
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Autocancel
- class pyflow.attributes.Autocancel(value, value2=None)
An attribute for automatic removal of the node which has completed.
- Parameters:
value (bool,int,str,list,tuple) – The time slot arguments for autocancel attribute. If True the node will be removed as soon as it has completed. If a list or tuple, items will be used for hour and minute when the node will be removed.
value2 (str) – The optional minute argument in case first argument contained only hour.
Example:
pyflow.attributes.Autocancel(True) # delete node immediately after completion pyflow.attributes.Autocancel('+01:30') # delete node 1 hour and 30 minutes after completion pyflow.attributes.Autocancel((1, 30)) # delete node at 1:30 am after completion pyflow.attributes.Autocancel(1, 30) # delete node at 1:30 am after completion pyflow.attributes.Autocancel(3) # delete node 3 days after completion
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Follow
- class pyflow.attributes.Follow(value)
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Parameters:
value (Repeat or Task or Family or Suite) – The followed node or the repeat attribute of the followed node.
- Example::
t1 = Task(“t1”, repeat=(RepeatEnumerated, “NUM”, [1, 2, 3])) t2 = Task(“t2”, repeat=(RepeatEnumerated, “NUM”, [1, 2, 3])) t2.follow = t1
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
Zombies
- class pyflow.attributes.Zombies(value)
An attribute that defines how a zombie should be handled in an automated fashion.
- Parameters:
value (str) – A custom way a zombie should be handled when encountered.
Example:
pyflow.attributes.Zombies(None)
- relative_path(other)
Returns relative path of the attribute.
- Parameters:
other (str) – Relative path of the parent node.
- Returns:
Relative path of the attribute.
- Return type:
str
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property fullname
The relative path of the attribute.
- Type:
str
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property value
The value of the node.
- Type:
str
External
- class pyflow.ExternSuite(path)
Maps an external suite.
- Parameters:
path (str) – Path of the external suite.
- Returns:
An object that corresponds to an external suite.
- Return type:
Example:
pyflow.ExternSuite('/a')
- class pyflow.ExternFamily(path)
Maps an external family, i.e. a family that is not built from the same repository.
- Parameters:
path (str) – Path of the external family.
- Returns:
An object that corresponds to an external family.
- Return type:
Example:
pyflow.ExternFamily('/f/g/h/i')
- class pyflow.ExternTask(path)
Maps an external task, i.e. a task that is not built from the same repository.
- Parameters:
path (str) – Path of the external task.
- Returns:
An object that corresponds to an external task.
- Return type:
Example:
pyflow.ExternTask('/a/b/c/d')
- class pyflow.ExternVariable(path)
Maps an external variable.
- Parameters:
path (str) – Path of the item.
- Returns:
An object that corresponds to an external variable.
- Return type:
Example:
pyflow.ExternVariable('/a/b:var')
- class pyflow.ExternLimit(path)
Maps an external limit.
- Parameters:
path (str) – Path of the item.
- Returns:
An object that corresponds to an external item.
- Return type:
Example:
pyflow.ExternLimit('/a/limits:hpc')
- class pyflow.ExternEvent(path)
Maps an external event, i.e. a event that is not built from the same repository.
- Parameters:
path (str) – Path of the external event.
- Returns:
An object that corresponds to an external event.
- Return type:
Example:
pyflow.ExternEvent('/e/f/g/h:ev')
- class pyflow.ExternMeter(path)
Maps an external meter, i.e. a meter that is not built from the same repository.
- Parameters:
path (str) – Path of the external meter.
- Returns:
An object that corresponds to an external event.
- Return type:
Example:
pyflow.ExternMeter('/g/h/i/j:mt')
- class pyflow.ExternRepeat(path)
Maps an external repeat, i.e. a repeat that is not built from the same repository. Cannot be a generic attribute as the repeat can be used with the follow() approach, which requires an object of type Repeat.
- Parameters:
path (str) – Path of the external repeat.
- Returns:
An object that corresponds to an external repeat.
- Return type:
Example:
pyflow.ExternRepeat('/a/b/c/d:YMD')
Deployment
- class pyflow.Notebook(suite, **options)
A dummy deployment target for Jupyter Notebooks, skips creation of directories and files and dumps fresh ecFlow definitions as cell output.
- Parameters:
suite (Suite) – The suite object to deploy.
path (str) – The path to Git repository.
Example:
s = pyflow.Suite('suite') s.deploy_suite(target=pyflow.Notebook)
- copy(source, target)
Deploys the task script to target path.
- Parameters:
source (str,bytes,list) – The task script to deploy.
target (str) – The deployment path.
- deploy_headers()
Installs all required header files.
- deploy_manual(deploy_path, full_script)
Deploys the manual to target path.
- Parameters:
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the manual.
- deploy_task(deploy_path, full_script, required_includes)
Deploys the task to target path.
- Parameters:
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the task.
required_includes (list) – The list of required header files.
- files_install_path()
str: Returns the files install path.
- save(source, target)
Deploys the task script to target path.
- Parameters:
source (str) – The path to task script.
target (str) – The deployment path.
- class pyflow.DeployGitRepo(suite, path=None)
A deployment target for Git repositories, clears target repository and dumps fresh ecFlow definitions.
- Parameters:
suite (Suite) – The suite object to deploy.
path (str) – The path to Git repository.
Example:
s = pf.Suite('suite') pyflow.DeployGitRepo(s, path='/path/to/git')
- check(target)
Check if the target should be deployed. Returns False if target has already been deployed, True otherwise.
- Parameters:
target (str) – The target path for deployment.
- Returns:
True if the target path is valid for deployment, False otherwise.
- Return type:
bool
- copy(source, target)
Deploys the task script to target path. This method contains functionality needed for all deployments. Should be called in super() by all derived classes.
- Parameters:
source (str) – The path to task script.
target (str) – The deployment path.
- deploy_headers()
Installs all required header files.
- deploy_manual(deploy_path, full_script)
Deploys the manual to target path.
- Parameters:
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the manual.
- deploy_task(deploy_path, full_script, required_includes)
Deploys the task to target path.
- Parameters:
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the task.
required_includes (list) – The list of required header files.
- files_install_path()
str: Returns the files install path.
- patch_path(path)
Patches the path so it includes the complete ecFlow path.
- Parameters:
path (str) – The path to patch.
- Returns:
The patched path.
- Return type:
str
- save(source, target)
Deploys the task script to target path. This method contains functionality needed for all deployments. Should be called in super() by all derived classes.
- Parameters:
source (str,bytes,list) – The task script to deploy.
target (str) – The deployment path.
Hosts
- class pyflow.Host(name, hostname=None, scratch_directory=None, log_directory=None, resources_directory=None, limit=None, extra_paths=None, extra_variables=None, environment_variables=None, module_source=None, modules=None, purge_modules=False, label_host=True, user='docs', ecflow_path=None, server_ecfvars=False, submit_arguments=None, workdir=None, trap_signals=None)
An abstract base class for host-related functionality.
- Parameters:
name (str) – The name of the host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths, or for login details. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
submit_arguments (dict) – A dictionary of arguments to pass to the scheduler when submitting jobs, which each key is a label that can be referenced when creating tasks with the Host instance.
workdir (str) – Work directory for every task executed within the Host instance, if not overriden for a Node.
trap_signals (list) – The list of signals to trap. A default list is used if not set.
Example:
class MyHost(Host): pass
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific cleanup script, always empty.
- Type:
list
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
- class pyflow.NullHost(**kwargs)
A dummy host object invisible to ecFlow, but still throws exceptions if pyflow attempts to create tasks inside it.
- Parameters:
hostname (str) – The hostname of the host, otherwise null will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
trap_signals (list) – The list of signals to trap. A default list is used if not set.
Example:
with pyflow.Suite('s', host=pf.NullHost()): pass
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Skips setting an exec_host label on nodes where this host is freshly set.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
The host-specific implementation of preamble script, always raises an error.
- Raises:
AttributeError – Constructing tasks under NullHost is invalid.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific implementation of cleanup script, always raises an error.
- Raises:
AttributeError – Constructing tasks under NullHost is invalid.
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
- class pyflow.LocalHost(name='localhost', **kwargs)
A host object that executes scripts directly on the ecFlow server.
- Parameters:
name (str) – The name of the host, localhost by default.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, the limit is 20 tasks.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
trap_signals (list) – The list of signals to trap. A default list is used if not set.
Example:
pyflow.LocalHost(purge_modules=True, modules=['mod1/123', '-mod2/321', 'mod3/33'])
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- copy_file_to(source_file, target_file)
Returns the script for copying a file to host.
- Parameters:
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns:
The script for copying a file to host.
- Return type:
str
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- run_simple_command(cmd)
Returns the command to run a simple command on this host.
- Parameters:
cmd (str) – A simple command to run.
- Returns:
The command to run a simple command.
- Return type:
str
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific cleanup script, always empty.
- Type:
list
- property job_cmd
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type:
str
- property kill_cmd
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type:
str
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
- class pyflow.SSHHost(name, user=None, indirect_host=None, indirect_user=None, **kwargs)
A host object that executes scripts on the ecFlow server via SSH protocol.
- Parameters:
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, the limit is 20 tasks.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
trap_signals (list) – The list of signals to trap. A default list is used if not set.
Example:
pyflow.SSHHost('dhs9999', user='max', scratch_directory='/data/a_mounted_filesystem/tmp')
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- copy_file_to(source_file, target_file)
Returns the script for copying a file to host.
- Parameters:
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns:
The script for copying a file to host.
- Return type:
str
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- run_simple_command(cmd)
Returns the command to run a simple command on this host.
- Parameters:
cmd (str) – A simple command to run.
- Returns:
The command to run a simple command.
- Return type:
str
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific cleanup script, always empty.
- Type:
list
- property job_cmd
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type:
str
- property kill_cmd
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type:
str
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
- class pyflow.SLURMHost(name, **kwargs)
A host object that executes scripts on the ecFlow server via Slurm job scheduling system.
- Parameters:
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
with pyflow.Suite('s', host=pyflow.SLURMHost('slurm_a')): pass
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- copy_file_to(source_file, target_file)
Returns the script for copying a file to host.
- Parameters:
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns:
The script for copying a file to host.
- Return type:
str
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
list: The host-specific implementation of preamble script.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- run_simple_command(cmd)
Returns the command to run a simple command on this host.
- Parameters:
cmd (str) – A simple command to run.
- Returns:
The command to run a simple command.
- Return type:
str
- script_submit_arguments(submit_arguments)
Returns list of script submit arguments.
- Parameters:
submit_arguments (dict) – A dictionary of script submit arguments.
- Returns:
The list of script submit arguments.
- Return type:
list
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific cleanup script.
- Type:
list
- property job_cmd
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type:
str
- property kill_cmd
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type:
str
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
- class pyflow.PBSHost(name, **kwargs)
A host object that executes scripts on the ecFlow server via batch server.
- Parameters:
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
with pyflow.Suite('s', host=pyflow.PBSHost('host_a')): pass
- add_to_limits(task)
Adds a task to be contained within a hosts assigned limit.
- Parameters:
task (Task) – The task to contain within the assigned limit.
- build_label()
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters:
replace (bool) – Whether to replace the currently computed limit.
- Raises:
DuplicateNodeError –
- copy_file_to(source_file, target_file)
Returns the script for copying a file to host.
- Parameters:
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns:
The script for copying a file to host.
- Return type:
str
- get_host_submit_arguments(label: str)
Returns the submit arguments for the given label.
- Parameters:
label (str) – The label to get the submit arguments for.
- Returns:
The submit arguments for the given label.
- Return type:
dict
- host_preamble(exit_hook=None)
list: The host-specific implementation of preamble script.
- job_preamble(exit_hook=None)
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)
Returns the host-specific error function for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns:
The error function script.
- Return type:
str
- preamble_init(ecflowpath)
Returns the host-specific preamble initialisation section for jobs.
- Parameters:
ecflowpath (str) – The path to ecFlow.
- Returns:
The preamble initialisation script.
- Return type:
str
- run_simple_command(cmd)
Returns the command to run a simple command on this host.
- Parameters:
cmd (str) – A simple command to run.
- Returns:
The command to run a simple command.
- Return type:
str
- script_submit_arguments(submit_arguments)
Returns list of script submit arguments.
- Parameters:
submit_arguments (dict) – A dictionary of script submit arguments.
- Returns:
The list of script submit arguments.
- Return type:
list
- update_node_attributes(options)
Updates the attributes of a node with the host-specific values.
Parameters: - options (dict): The options dictionary to update with host-specific attributes.
- property check_cmd
The ecflow check command.
- Type:
str
- property host_postamble
The host-specific cleanup script.
- Type:
list
- property job_cmd
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type:
str
- property kill_cmd
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type:
str
- property limit
The number of tasks that can run on this host simultaneously.
- Type:
int
- property status_cmd
The ecflow status command.
- Type:
str
Scripts
- class pyflow.Script(value=None)
A group of commands that define the main work that is to be carried out for a task.
- Parameters:
value (str,list) – The script command or the list of script commands.
Example:
with pyflow.Task('t', script=pyflow.Script('echo "Hello, world!"')): pass
- static generate_list_scripts(values)
Generates a script of all passed scripts.
- Parameters:
values (list) – The list of Script objects.
- Returns:
The list of script commands.
- Return type:
list
- add_required_exportables(*args)
Defines a set of required exportable variables.
- Parameters:
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)
Defines a required exportable attribute.
- Parameters:
exportable (str) – The name of the exportable variables.
- generate_stub()
Returns complete script by combining the fragments.
- Returns:
The list of script commands.
- Return type:
list
- required_exportables()
Returns the set of required exportable variables.
- Returns:
The set of required exportable variables.
- Return type:
set
- property value
The string value with all script commands.
- Type:
str
- class pyflow.PythonScript(value=None, cmd_args={}, python=3)
A script written in Python language.
- Parameters:
value (str,list) – The Python command or the list of Python commands.
python (int) – The Python major version.
Example:
pyflow.PythonScript('w1 = "Hello"\nw2 = "world"\nprint(f"{w1}, {w2}!")', 3)
- static generate_list_scripts(values)
Generates a script of all passed scripts.
- Parameters:
values (list) – The list of Script objects.
- Returns:
The list of script commands.
- Return type:
list
- add_required_exportables(*args)
Defines a set of required exportable variables.
- Parameters:
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)
Defines a required exportable attribute.
- Parameters:
exportable (str) – The name of the exportable variables.
- generate_stub()
Returns complete script by combining the fragments.
- Returns:
The list of script commands.
- Return type:
list
- required_exportables()
Returns the set of required exportable variables.
- Returns:
The set of required exportable variables.
- Return type:
set
- property value
The string value with all script commands.
- Type:
str
- class pyflow.FileScript(filename)
A script with a provided filename, which is read at the time of suite generation.
- Parameters:
filename (str) – The filename of the script to read from.
Example:
pyflow.FileScript('/path/to/script')
- static generate_list_scripts(values)
Generates a script of all passed scripts.
- Parameters:
values (list) – The list of Script objects.
- Returns:
The list of script commands.
- Return type:
list
- add_required_exportables(*args)
Defines a set of required exportable variables.
- Parameters:
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)
Defines a required exportable attribute.
- Parameters:
exportable (str) – The name of the exportable variables.
- generate()
Returns the script commands from the provided file.
- Returns:
The list of script commands.
- Return type:
list
- generate_stub()
Returns complete script by combining the fragments.
- Returns:
The list of script commands.
- Return type:
list
- required_exportables()
Returns the set of required exportable variables.
- Returns:
The set of required exportable variables.
- Return type:
set
- property value
The string value with all script commands.
- Type:
str
- class pyflow.TemplateScript(*args, **kwargs)
A script template with Jinja syntax support.
- Parameters:
value (str,list) – The script command or the list of script commands.
**kwargs (dict) – Accept keyword arguments as values for template variables.
Example:
pyflow.TemplateScript('echo "{{ w1 }}, {{ w2 }}!"', w1='Hello', w2='world')
- static generate_list_scripts(values)
Generates a script of all passed scripts.
- Parameters:
values (list) – The list of Script objects.
- Returns:
The list of script commands.
- Return type:
list
- add_parameters(**kwargs)
Defines extra parameters for template variables.
- Parameters:
**kwargs (dict) – Accept keyword arguments as values for template variables.
- add_required_exportables(*args)
Defines a set of required exportable variables.
- Parameters:
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)
Defines a required exportable attribute.
- Parameters:
exportable (str) – The name of the exportable variables.
- generate_stub()
Returns complete script by combining the fragments.
- Returns:
The list of script commands.
- Return type:
list
- required_exportables()
Returns the set of required exportable variables.
- Returns:
The set of required exportable variables.
- Return type:
set
- property value
The string value with all script commands.
- Type:
str
- class pyflow.TemplateFileScript(*args, **kwargs)
A script template with a provided filename, which is read at the time of suite generation.
- Parameters:
filename (str) – The filename of the script to read from.
**kwargs (dict) – Accept keyword arguments as values for template variables.
Example:
pyflow.TemplateFileScript('/path/to/script', foo='bar')
- static generate_list_scripts(values)
Generates a script of all passed scripts.
- Parameters:
values (list) – The list of Script objects.
- Returns:
The list of script commands.
- Return type:
list
- add_parameters(**kwargs)
Defines extra parameters for template variables.
- Parameters:
**kwargs (dict) – Accept keyword arguments as values for template variables.
- add_required_exportables(*args)
Defines a set of required exportable variables.
- Parameters:
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)
Defines an environment variable for the script.
- Parameters:
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)
Defines a required exportable attribute.
- Parameters:
exportable (str) – The name of the exportable variables.
- generate()
Returns the script commands from the provided file.
- Returns:
The list of script commands.
- Return type:
list
- generate_stub()
Returns complete script by combining the fragments.
- Returns:
The list of script commands.
- Return type:
list
- required_exportables()
Returns the set of required exportable variables.
- Returns:
The set of required exportable variables.
- Return type:
set
- property value
The string value with all script commands.
- Type:
str
Resources
- class pyflow.DataResource(name, hosts, source_data)
Provides a data resource to be deployed at suite generation time.
- Parameters:
name (str) – The name of the resource.
hosts (Host,list) – The host or list of hosts to deploy the resource to.
source_data – The resource data.
Example:
pyflow.DataResource('data1', [pyflow.LocalHost(resources_directory='/path/to/data')], "this is some data".encode('utf-8'))
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow task object.
- Returns:
ecFlow task object.
- Return type:
ecflow.Task
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_script()
Returns the installer script for the data resource.
- Returns:
The list of installer commands.
- Return type:
list
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- location()
Returns the path of the resource.
- Returns:
The path of the resource.
- Return type:
str
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- md5()
Returns MD5 checksum of the resource data.
- Returns:
The MD5 checksum of the resource data.
- Return type:
str
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- save_data(target, filename)
Deploys the data for the resource at suite generation time.
- Parameters:
target (Deployment) – The deployment target.
filename (str) – The filename for the resource data.
- task_modules()
Returns list of task modules.
- Returns:
List of task modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the host or task modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property deploy_extension
The script file extension to be used during deployment of the task.
- Type:
str
- property deploy_path
The deployment path of the current task, may be None.
- Type:
str
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submit_arguments
The dictionary of submit arguments.
- Type:
dict
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
- class pyflow.FileResource(name, hosts, source_file)
Provides a file resource to be deployed at suite generation time.
- Parameters:
name (str) – The name of the resource.
hosts (list,Host_) – The list of hosts to deploy the resource to.
source_file (str) – The filename of the resource.
Example:
pyflow.FileResource('data2', [pyflow.LocalHost(resources_directory='/path/to/data')], 'path/to/data.dat')
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- data()
Returns the resource data from the provided file.
- Returns:
The resource data.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow task object.
- Returns:
ecFlow task object.
- Return type:
ecflow.Task
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_script()
Returns the installer script for the data resource.
- Returns:
The list of installer commands.
- Return type:
list
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- location()
Returns the path of the resource.
- Returns:
The path of the resource.
- Return type:
str
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- md5()
Returns MD5 checksum of the resource data.
- Returns:
The MD5 checksum of the resource data.
- Return type:
str
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- save_data(target, filename)
Deploys the data for the resource at suite generation time.
- Parameters:
target (Deployment) – The deployment target.
filename (str) – The filename for the resource data.
- task_modules()
Returns list of task modules.
- Returns:
List of task modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the host or task modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property deploy_extension
The script file extension to be used during deployment of the task.
- Type:
str
- property deploy_path
The deployment path of the current task, may be None.
- Type:
str
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submit_arguments
The dictionary of submit arguments.
- Type:
dict
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
- class pyflow.WebResource(name, hosts, url, md5=None)
Provides a web resource to be deployed at suite generation time.
- Parameters:
name (str) – The name of the resource.
hosts (list,Host_) – The list of hosts to deploy the resource to.
url (str) – The URL of the resource.
md5 (str) – The MD5 checksum of the resource.
Example:
pyflow.WebResource('data3', [pyflow.LocalHost(resources_directory='/path/to/data')], 'https://example.com/data', md5='0123456789abcdef')
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow task object.
- Returns:
ecFlow task object.
- Return type:
ecflow.Task
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_script()
Returns the installer script for the data resource.
- Returns:
The list of installer commands.
- Return type:
list
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- get_resource(filename)
Returns script commands to retrieve the resource to deploy at runtime.
- Parameters:
filename (str) – The filename for the resource data.
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- location()
Returns the path of the resource.
- Returns:
The path of the resource.
- Return type:
str
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- md5()
Returns MD5 checksum of the resource data. If not provided at construction time, resource will be downloaded and hashed.
- Returns:
The MD5 checksum of the resource data.
- Return type:
str
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of task modules.
- Returns:
List of task modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the host or task modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property deploy_extension
The script file extension to be used during deployment of the task.
- Type:
str
- property deploy_path
The deployment path of the current task, may be None.
- Type:
str
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submit_arguments
The dictionary of submit arguments.
- Type:
dict
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
- class pyflow.Resources(name=None, host=None)
Provides both visual and logical grouping of related resources.
Note
Just a wrapper (for now) on Family, used for consistency and elegance.
- Parameters:
name (str) – The name of the resource group to create.
host (Host) – The host to execute the resource group on.
Example:
with pyflow.Resources(host=pf.LocalHost()): pass
- add_node(node)
Adds a child to current node.
- Parameters:
node (Node) – The child node to add.
- Returns:
Added child node.
- Return type:
Node
- append_node(node)
Appends a child to current node.
- Parameters:
node (Node) – The child node to append.
- Returns:
The parent node.
- Return type:
Node
- check_definition()
Checks ecFlow definitions of the node.
- Raises:
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)
Removes child nodes of provided type.
- Parameters:
cls (class) – The node type class name.
- draw_graph(view=True)
Draws the DOT graph.
- Parameters:
view (bool) – Unused.
- Returns:
DOT graph.
- Return type:
Dot
- draw_tree()
Draws node tree as a DOT graph.
- Returns:
The node DOT graph.
- Return type:
Dot
- ecflow_definition()
Returns node definition.
- Returns:
The node definition.
- Return type:
ecflow.Defs
- ecflow_object()
Returns the corresponding ecFlow family object.
- Returns:
ecFlow family object.
- Return type:
ecflow.Family
- find_node(subpath)
Returns node under provided path.
- Parameters:
subpath (str) – Path of the node to search for.
- Returns:
The found node object.
- Return type:
Node
- generate_node()
Generates node definition.
- Returns:
Generated ecFlow node object.
- Return type:
Node
- generate_stub(scripts)
Returns complete script by combining the fragments.
- Parameters:
scripts (tuple) – List of script fragments.
- Returns:
Complete script.
- Return type:
str
- has_variable(name)
Signals if the current node has a variable defined.
- Parameters:
name (str) – Name of the variable to search for.
- Returns:
Whether the current node has the variable defined or not.
- Return type:
bool
- lookup_variable(name)
Looks up value of the variable in current or parent node.
- Parameters:
name (str) – Name of the variable to look up for.
- Returns:
Variable value, if found.
- Return type:
str
- lookup_variable_value(name, default=None)
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters:
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns:
Variable value, if found. Otherwise provided default value.
- Return type:
str
- make_expression()
Generates node expression.
- Returns:
Whether the node is complete or not.
- Return type:
bool
- relative_path(node)
Returns relative path of the node.
- Returns:
Relative path of the node.
- Return type:
str
- remove_node(node)
Removes specific child node.
- Parameters:
node (Node) – The child node to remove.
- replace_on_server(host, port=None)
Replaces node on the target host.
- Parameters:
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()
Returns list of modules.
- Returns:
List of modules.
- Return type:
list
- task_purge_modules()
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns:
Whether the modules have been purged or not.
- Return type:
bool
- property aborted
The node aborted status.
- Type:
bool
- property active
The node active status.
- Type:
bool
- property all_exportables
The dictionary of all exportable attributes in the current or parent node.
- Type:
dict
- property all_variables
The dictionary of all variables in the current or parent node.
- Type:
dict
- property anchor
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type:
Anchor
- property autocancel
An attribute for automatic removal of the node which has completed.
- Type:
- property children
The list of all direct child nodes.
- Type:
list
- property complete
The node complete status.
- Type:
bool
- property completes
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type:
- property events
An attribute for declaring an action that a node can trigger while it is running.
- Type:
- property executable_children
The list of all tasks and families directly contained within a Family.
- Type:
list
- property follow
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type:
- property fullname
The full path of the node from the root.
- Type:
str
- property generated_variables
An attribute for referencing an ecFlow generated variable.
- Type:
- property headers
The current and parent node headers, including head and tail.
- Type:
list
- property host
Returns the currently active host object. If not found in current node, search in parents.
- Returns:
Currently active host object.
- Return type:
- property inlimits
An attribute for grouping of nodes to which a limit should be applied.
- Type:
- property limits
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type:
- property manual
The manual of the node, i.e. help text.
- Type:
str,list
- property manual_path
The deployment path of the current task, may be None.
- Type:
str
- property meters
An attribute for a range of integer values that can be set from a script.
- Type:
- property mirror
An attribute that allows a node status to be synchronized with a node from another ecFlow server.
- Type:
- property modules
The list of environment modules for the node.
- Type:
list
- property name
The visible name of the node.
- Type:
str
- property parent
The parent node object.
- Type:
Node
- property path_list
The list of node paths.
- Type:
list
- property queued
The node queued status.
- Type:
bool
- property repeat
Returns the currently active repeat object. If not found in current node, search in parents.
- Returns:
Currently active repeat object.
- Return type:
- property submitted
The node submitted status.
- Type:
bool
- property to_html
The representation of the node in HTML.
- Type:
str
- property today
An attribute for setting a cron dependency of the node for the current day.
- Type:
- property triggers
An attribute for setting a condition for running the node depending on other tasks or families.
- Type:
- property unknown
The node unknown status.
- Type:
bool
- property value
The value of the node.
- Type:
str
- property workdir
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns:
Currently active working directory for tasks.
- Return type:
string
Miscellaneous
State
- pyflow.state.aborted = ecflow.ecflow.DState.aborted
Defines an aborted state.
- pyflow.state.active = ecflow.ecflow.DState.active
Defines an active state.
- pyflow.state.complete = ecflow.ecflow.DState.complete
Defines a complete state.
- pyflow.state.queued = ecflow.ecflow.DState.queued
Defines a queued state.
- pyflow.state.submitted = ecflow.ecflow.DState.submitted
Defines a submitted state.
- pyflow.state.suspended = ecflow.ecflow.DState.suspended
Defines a suspended state.
- pyflow.state.unknown = ecflow.ecflow.DState.unknown
Defines an unknown state.
Helper Functions
- pyflow.ecflow_name(s)
Converts a string value to one that can be used in names of ecFlow objects, by removing or converting illegal characters.
- Parameters:
s (str) – A string value to convert.
- Returns:
The string value safe for use in names of ecFlow objects.
- Return type:
str
Example:
pyflow.ecflow_name('hyphenated-name')
- pyflow.all_complete(nodes)
Returns a trigger expression for all of the supplied nodes being complete.
- Parameters:
nodes (list) – The list of input nodes.
- Returns:
Trigger expression for all of the supplied nodes being complete.
- Return type:
expression
Example:
trigger = pyflow.all_complete(pyflow.Task('task-{}'.format(i)) for i in range(10))
- pyflow.sequence(nodes)
Sets triggers so that the input tasks/families will run in sequence, in the order that they are given.
The input nodes are modified in-place. Existing triggers will be preserved by &-ing them with the new trigger.
- Parameters:
nodes (list) – The list of input nodes.
Example:
pyflow.sequence(pyflow.Task('{}_{}'.format(name, i), i) for i in range(counters))
Deferred
- class pyflow.Deferred(func, *args)
Defines an expression which will be evaluated only at suite generation time.
- Parameters:
func (function) – The callback function to generate the expression.
*args (tuple) – Accept extra positional arguments for the callback function.
Example:
def my_callback(var, val): pass pyflow.Deferred(my_callback, 'foo', 'bar')
- generate_expression(parent=None)
Generates the expression.
- Parameters:
parent (Node) – The parent node to generate the expression for.
- Returns:
Generated expression.
- Return type:
expression