Configuring Suites
We build such library of classes and objects so we can re-use these components (Tasks, Families, Suites) in different contexts. A given task class could be used in a research workflow and then reused in another operational workflow.
However different contexts may require some differences in the suite execution. To ensure that we still have a concise, maintainable and easily checkable suite, we need to cater for those differences preferably in a single entity (as opposed to spreadout through the suite).
To that aim, we introduce the use of a configuration object that will handle the differences, and therefore interact and configure our objects under each different context.
This results in suites that are configurable for different use-cases and different contexts and build fundamentally different generated suites from the same components
A configuration object can be constructed manually for different use cases or as a result of parsing configuration files. It can be used to:
Provide constants and data for specific cases, that will be needed in the suites.
Switch functionality on/off or modify it.
Configuration for hosts where to run the tasks.
Locations of and details of data to process.
But most importantly, as objects, these configuration objects can be programmable in themselves (can include code). The suite components can delegate part of the suite definition to these configurators and as such the structure of the suite can be determined by logic in the configuration object if necessary.
Important
Delegation is preferred over conditional if statements in the suite depending on configuration values.
[2]:
class BaseConfig:
"""This is a very contrived example showing delegation of behaviour to configuration"""
def __init__(self, name, common_count=3, unit_count=4, integration_count=5):
self.name = name
self.common_count=common_count
self.unit_count = unit_count
self.integration_count = integration_count
def build_unit_tests(self):
pass
def build_integration_tests(self):
pass
class ProductionConfig(BaseConfig):
def build_integration_tests(self):
with pf.Family('integration') as f:
pf.sequence(MyTask('integration_{}'.format(i), 123*i) for i in range(self.integration_count))
return f
class DevConfig(BaseConfig):
def build_unit_tests(self):
with pf.Family('unit') as f:
pf.sequence(MyTask('unit_{}'.format(i), 123*i) for i in range(self.unit_count))
return f
We can now build a common testing family that behaves (structurally) differently according to the configuration supplied.
[3]:
class ConfiguredFamily(pf.Family):
def __init__(self, config):
super().__init__(config.name)
with self:
# the static part of the suite, common to all suites of this type
with pf.Family('common') as common:
pf.sequence(MyTask('common_{}'.format(i), 123*i) for i in range(5))
# the dynamic part of the suite, with hooks for the variability
test_families = [
config.build_unit_tests(),
config.build_integration_tests()
]
# some other static of the suite
with pf.Family('cleanup') as cleanup:
MyTask('cleaner')
# establish dependencies
common >> cleanup
for f in test_families:
if f is not None:
common >> f >> cleanup
[4]:
with CourseSuite('configuration_example') as s:
ConfiguredFamily(ProductionConfig('prod', integration_count=3))
ConfiguredFamily(DevConfig('dev', unit_count=25))
s
[4]:
suite configuration_example defstatus suspended edit ECF_FILES '/path/to/scratch/files/configuration_example' edit ECF_HOME '/path/to/scratch/out' edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &' edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%' edit ECF_STATUS_CMD 'true' edit ECF_OUT '%ECF_HOME%' label exec_host "localhost" family prod family common task common_0 edit HALF '0' edit LIMIT '0' label counter_label "count to 0" task common_1 trigger common_0 eq complete edit HALF '123' edit LIMIT '246' label counter_label "count to 246" task common_2 trigger common_1 eq complete edit HALF '246' edit LIMIT '492' label counter_label "count to 492" task common_3 trigger common_2 eq complete edit HALF '369' edit LIMIT '738' label counter_label "count to 738" task common_4 trigger common_3 eq complete edit HALF '492' edit LIMIT '984' label counter_label "count to 984" endfamily family integration trigger common eq complete task integration_0 edit HALF '0' edit LIMIT '0' label counter_label "count to 0" task integration_1 trigger integration_0 eq complete edit HALF '123' edit LIMIT '246' label counter_label "count to 246" task integration_2 trigger integration_1 eq complete edit HALF '246' edit LIMIT '492' label counter_label "count to 492" endfamily family cleanup trigger common eq complete and integration eq complete task cleaner edit HALF '0' edit LIMIT '0' label counter_label "count to 0" endfamily endfamily family dev family common task common_0 edit HALF '0' edit LIMIT '0' label counter_label "count to 0" task common_1 trigger common_0 eq complete edit HALF '123' edit LIMIT '246' label counter_label "count to 246" task common_2 trigger common_1 eq complete edit HALF '246' edit LIMIT '492' label counter_label "count to 492" task common_3 trigger common_2 eq complete edit HALF '369' edit LIMIT '738' label counter_label "count to 738" task common_4 trigger common_3 eq complete edit HALF '492' edit LIMIT '984' label counter_label "count to 984" endfamily family unit trigger common eq complete task unit_0 edit HALF '0' edit LIMIT '0' label counter_label "count to 0" task unit_1 trigger unit_0 eq complete edit HALF '123' edit LIMIT '246' label counter_label "count to 246" task unit_2 trigger unit_1 eq complete edit HALF '246' edit LIMIT '492' label counter_label "count to 492" task unit_3 trigger unit_2 eq complete edit HALF '369' edit LIMIT '738' label counter_label "count to 738" task unit_4 trigger unit_3 eq complete edit HALF '492' edit LIMIT '984' label counter_label "count to 984" task unit_5 trigger unit_4 eq complete edit HALF '615' edit LIMIT '1230' label counter_label "count to 1230" task unit_6 trigger unit_5 eq complete edit HALF '738' edit LIMIT '1476' label counter_label "count to 1476" task unit_7 trigger unit_6 eq complete edit HALF '861' edit LIMIT '1722' label counter_label "count to 1722" task unit_8 trigger unit_7 eq complete edit HALF '984' edit LIMIT '1968' label counter_label "count to 1968" task unit_9 trigger unit_8 eq complete edit HALF '1107' edit LIMIT '2214' label counter_label "count to 2214" task unit_10 trigger unit_9 eq complete edit HALF '1230' edit LIMIT '2460' label counter_label "count to 2460" task unit_11 trigger unit_10 eq complete edit HALF '1353' edit LIMIT '2706' label counter_label "count to 2706" task unit_12 trigger unit_11 eq complete edit HALF '1476' edit LIMIT '2952' label counter_label "count to 2952" task unit_13 trigger unit_12 eq complete edit HALF '1599' edit LIMIT '3198' label counter_label "count to 3198" task unit_14 trigger unit_13 eq complete edit HALF '1722' edit LIMIT '3444' label counter_label "count to 3444" task unit_15 trigger unit_14 eq complete edit HALF '1845' edit LIMIT '3690' label counter_label "count to 3690" task unit_16 trigger unit_15 eq complete edit HALF '1968' edit LIMIT '3936' label counter_label "count to 3936" task unit_17 trigger unit_16 eq complete edit HALF '2091' edit LIMIT '4182' label counter_label "count to 4182" task unit_18 trigger unit_17 eq complete edit HALF '2214' edit LIMIT '4428' label counter_label "count to 4428" task unit_19 trigger unit_18 eq complete edit HALF '2337' edit LIMIT '4674' label counter_label "count to 4674" task unit_20 trigger unit_19 eq complete edit HALF '2460' edit LIMIT '4920' label counter_label "count to 4920" task unit_21 trigger unit_20 eq complete edit HALF '2583' edit LIMIT '5166' label counter_label "count to 5166" task unit_22 trigger unit_21 eq complete edit HALF '2706' edit LIMIT '5412' label counter_label "count to 5412" task unit_23 trigger unit_22 eq complete edit HALF '2829' edit LIMIT '5658' label counter_label "count to 5658" task unit_24 trigger unit_23 eq complete edit HALF '2952' edit LIMIT '5904' label counter_label "count to 5904" endfamily family cleanup trigger common eq complete and unit eq complete task cleaner edit HALF '0' edit LIMIT '0' label counter_label "count to 0" endfamily endfamily endsuite