Advanced Attributes
Creation of Attributes
Typically, we have three methods to construct attributes (or sub nodes) attached to any specific node. We give here examples both within a simple tree formulation of a suite, or within a class derived from a specific pyflow class.
These different methods have different constraints on them, and differ in clarity and legibility in different contexts. Ultimately, the choice of which to use should come down to which is most legible in context.
Firstly, we can construct the pyflow object within a context manager containing the parent node.
[2]:
with pf.Suite('s', host=pf.NullHost()) as s:
with pf.Family('f') as f:
pf.Label('l', 'text')
pf.Variable('V', 'value')
s
[2]:
suite s family f edit V 'value' label l "text" endfamily endsuite
[3]:
class DerivedFamily(pf.Family):
def __init__(self):
super().__init__('f')
with self:
pf.Label('l', 'text')
pf.Variable('V', 'value')
with pf.Suite('s', host=pf.NullHost()) as s:
DerivedFamily()
s
[3]:
suite s family f edit V 'value' label l "text" endfamily endsuite
Secondly, objects can be allocated by using keyword arguments on the parent node constructor. These take three forms:
For an attribute of which there can only be one instance, the keyword argument is the lower-case string of the attribute class name. E.g.
script=.For an attribute of which there cane be multiple instances, the keyword argument is the lower-case, pluralised version of the class name. E.g.
labels=, and accepts a list or tuple.ecFlow variables are passed in as direct keyword arguments, identified by being capitalised and valid ecFlow variable names.
[4]:
with pf.Suite('s', host=pf.NullHost()) as s:
pf.Family('f', labels={'l': 'text'}, V='value')
s
[4]:
suite s family f edit V 'value' label l "text" endfamily endsuite
[5]:
class DerivedFamily(pf.Family):
def __init__(self, **kwargs):
variables = {'V': 'value'}
variables.update(kwargs)
labels = {'l': 'text'}
super().__init__('f', labels=labels, **variables)
with pf.Suite('s', host=pf.NullHost()) as s:
DerivedFamily()
s
[5]:
suite s family f edit V 'value' label l "text" endfamily endsuite
Finally, unambiguously named pyflow objects (variables, script, …) can be directly assigned to their parent nodes.
[6]:
with pf.Suite('s', host=pf.NullHost()) as s:
f = pf.Family('f')
f.V = 'value'
s
[6]:
suite s family f edit V 'value' endfamily endsuite
Best Practice for Variables and Attributes
Best practice for pyflow is to create derived types that encapsulate all of the concerns of a given class. This means that variable and attribute creation should occur within the constructor of the class being written. This should generally take the form of a setup section, in which various children are defined, before passing them through to the constructor of the superclass. Any structural children should then be defined below.
[7]:
class ExampleFamily(pf.Family):
def __init__(self, name, example_value, initial_label, **kwargs):
# This structure allows the kwargs to override any of these variables if needed, or
# to set other more general properties of the superclass (such as host=). The same
# effect could be achieved by using kwargs.setdefault(...) and passing kwargs through.
variables = {
'REQUIRED_VARIABLE': 'required_value',
'EXAMPLE_VARIABLE': example_value
}
variables.update(kwargs)
labels = {
'a_label': initial_label
}
super().__init__(name, labels=labels, **variables)
# Here we define structural children
with self:
(
MyFamily('f1')
>>
MyTask('t1')
)
Variable substitition and expansion
Variables and attributes can be directly referred to in scripts by making use of automatically exported environment variables of the same name. For example, a RepeatDate('YMD', ...) object may be referred to in a script by writing $YMD. This will be automatically detected by pyflow and the variable exported.
If generating scripts, or using the templating engine, pyflow objects can generate their own representations. The str() and repr() functions in Python will return representations of variables that can be used in scripts (after automatic variable exporting) and in technical contexts (pre variable exporting, such as in other ecFlow variables) respectively.
We can access the properties of an ecflow Variable programatically. This allows us to make interdependencies explicit, and to generate snippets within scripts that are guaranteed to correctly use the objects.
[8]:
with pf.Suite('s'):
v = pf.Variable('A_VARIABLE', 1234)
print(str(v), repr(v), v.value)
print(v.name, v.fullname)
$A_VARIABLE %A_VARIABLE% 1234
A_VARIABLE /s:A_VARIABLE
This allows us to automatically generate the correct shell-expansion of variables in the appropriate script context. Note that both Python string substitution and Jinja2 templating use the str() representation by default.
[9]:
text_script = 'echo "Variable value: {}"'.format(v)
print(text_script)
echo "Variable value: $A_VARIABLE"
[10]:
templated_script = pf.TemplateScript(
'echo "variable {{ VARIABLE.name }} has value {{ VARIABLE }}"',
VARIABLE=v
)
print(templated_script)
echo "variable A_VARIABLE has value $A_VARIABLE"
Other ecFlow objects that set accessible values can be accessed in the same way.
[11]:
with pf.Suite('s') as s:
pf.RepeatDate("YMD", datetime.date(2019, 1, 1), datetime.date(2019, 12, 31))
print(pf.TemplateScript(
'echo "The current date object is {{ YMD.name }}. Value={{ YMD }}',
YMD=s.YMD
))
echo "The current date object is YMD. Value=$YMD
We can also use templating to facilitate accessing attributes using the ecflow_client, and to correctly set thew according to mutable values (including ecFlow variables).
[12]:
with pf.Suite('s', FOO='bar') as s:
pf.Label('label', '')
print(pf.TemplateScript(
'ecflow_client --alter=change label {{ LABEL.name }} "{{ VALUE }}" {{ LABEL.parent.fullname }}',
LABEL=s.label,
VALUE=s.FOO
))
ecflow_client --alter=change label label "$FOO" /s
Using attributes belonging to other nodes
Attributes associated with other nodes can be used by passing the relevant attribute object to the site where it is needed. This can be facilitated by accessing children of various nodes as attributes of the parent.
[13]:
with pf.Suite('s') as s:
with pf.Family('family1') as f1:
pf.Label('the_label', '')
with pf.Family('family2') as f2:
LabelSetter((f1.the_label, "a value"), name='labeller')
print(f2.labeller.script)
ecflow_client --alter=change label the_label "a value" /s/family1
In contexts where the relative path between nodes and attributes is required, the relative_path method is able to interrogate the relationships. Alternatively the fullname attribute will give the absolute path of nodes.
Within pyflow expressions it should not be necessary to generate these paths manually, as the expression generator should do the right thing. However, it is sometimes useful to refer to these components within scripts, especially as expansions within templates scripts.
[14]:
print(s.family1.the_label.relative_path(s.family2))
print(s.family2.labeller.relative_path(s.family1))
print(s.family2.labeller.relative_path(s.family1.the_label))
print(s.family2.labeller.fullname)
print(s.family1.the_label.fullname)
print('\nscript: \n', pf.TemplateScript(
'location of external node: {{ NODE.fullname }}',
NODE=s.family2.labeller
))
print('\nscript: \n', pf.TemplateScript(
'attribute relative path: {{ ATTRIBUTE.relative_path(NODE) }}',
ATTRIBUTE=s.family1.the_label,
NODE=s.family2.labeller
))
family1:the_label
family2/labeller
../family2/labeller
/s/family2/labeller
/s/family1:the_label
script:
location of external node: /s/family2/labeller
script:
attribute relative path: ../family1:the_label
Using variables defined in parents
ecFlow suites inherit variables from above. If a task is making use of these variables it is very easy to end up writing tasks that assume the existence of variables in a suite already, without anything programattically indicating or enforcing that this relationship exists.
Derived Tasks that make use of external variables should require that they be passed in from outside. If they are not directly used (i.e. the value is used in the script directly) then validity should be asserted in the code.
[15]:
class ChildTask(pf.Task):
def __init__(self, external_variable):
assert external_variable.name == 'EXTERNAL_VAR'
script = 'echo "external variable: $EXTERNAL_VAR"'
super().__init__('uses_var', script=script)
with CourseSuite('assert_external_variable') as s:
with pf.Family('containing_family', EXTERNAL_VAR=1234) as f:
ChildTask(f.EXTERNAL_VAR)
s
[15]:
suite assert_external_variable defstatus suspended edit ECF_FILES '/path/to/scratch/files/assert_external_variable' edit ECF_HOME '/path/to/scratch/out' edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &' edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%' edit ECF_STATUS_CMD 'true' edit ECF_OUT '%ECF_HOME%' label exec_host "localhost" family containing_family edit EXTERNAL_VAR '1234' task uses_var endfamily endsuite
[16]:
print("script:\n", f.uses_var.script, '\n')
script:
echo "external variable: $EXTERNAL_VAR"
If scripts are being generated or templated, then the existence of inherited variables can be enforced through generation.
[17]:
class ChildTask(pf.Task):
def __init__(self, external_variable):
script = pf.TemplateScript(
'echo "external variable: {{ VARIABLE }}"',
VARIABLE=external_variable
)
super().__init__('uses_var', script=script)
with CourseSuite('templated_external_variable') as s:
with pf.Family('containing_family', MY_VAR=1234) as f:
ChildTask(f.MY_VAR)
s
[17]:
suite templated_external_variable defstatus suspended edit ECF_FILES '/path/to/scratch/files/templated_external_variable' edit ECF_HOME '/path/to/scratch/out' edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &' edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%' edit ECF_STATUS_CMD 'true' edit ECF_OUT '%ECF_HOME%' label exec_host "localhost" family containing_family edit MY_VAR '1234' task uses_var endfamily endsuite
[18]:
print("script:\n", f.uses_var.script, '\n')
script:
echo "external variable: $MY_VAR"
Alternatively, we can provide default values which are overridden in the context of an externally supplied variable.
[19]:
class TaskWithVariable(pf.Task):
def __init__(self, name, default_value=1234, **kwargs):
super().__init__(name, **kwargs)
# Note that this sort of introspective setup is one that requires constructing
# components after calling the superclass
if isinstance(default_value, pf.Variable):
var = default_value
else:
self.TASK_VALUE = default_value
var = self.TASK_VALUE
self.script = pf.TemplateScript(
'echo "external variable: {{ VARIABLE }}"',
VARIABLE=var
)
with CourseSuite('internal_or_external_variable') as s:
with pf.Family('containing_family', MY_VAR=1234) as f:
TaskWithVariable('external_variable', f.MY_VAR)
TaskWithVariable('external_value', f.MY_VAR.value)
TaskWithVariable('default_value')
s
[19]:
suite internal_or_external_variable defstatus suspended edit ECF_FILES '/path/to/scratch/files/internal_or_external_variable' edit ECF_HOME '/path/to/scratch/out' edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &' edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%' edit ECF_STATUS_CMD 'true' edit ECF_OUT '%ECF_HOME%' label exec_host "localhost" family containing_family edit MY_VAR '1234' task external_variable task external_value edit TASK_VALUE '1234' task default_value edit TASK_VALUE '1234' endfamily endsuite
[20]:
print("script external:\n", f.external_variable.script, '\n')
print("script default:\n", f.default_value.script, '\n')
script external:
echo "external variable: $MY_VAR"
script default:
echo "external variable: $TASK_VALUE"
General node properties
Nodes and attributes have many accessible properties that can be accessed. Here is a non-exhaustive list of useful general node properties:
suite- TheSuiteobject containing the nodehost()- The currently activeHostobjectanchor- The current anchor (eitherSuiteorAnchorFamily) containing this nodename- The visible name of this nodefullname- The full path of this node from the rootall_children- All (direct) children of a nodeall_executable_children- AllTasksandFamilies(directly) contained within aFamilyall_tasks- AllTasks(directly) contained within aFamily