Helper Functionality

pyflow aims to contain not just a collection of ecflow functionality, but also helper functionality to assist in building suites. Where idiomatic uses of ecFlow result in the same mechanisms being built repeatedly, pyflow can incorporate these to help generate clearer suites.

The ecflow_name() functionality converts an arbitrary string into a name which meets the character restrictions for ecflow nodes. This is very useful for converting strings such as hostnames or the names of various data sets into a form that can be used as the name of a Family or Task.

print(pf.ecflow_name('hyphenated-name'))

hyphenated_name

The all_complete() and sequence() functions facilitate working with generated sequences of python tasks. all_complete() generates an expression suitable for use in triggers (or completes). sequence() generates triggers such that all of the tasks will run sequentially.

with CourseSuite('sequences') as s:
    tasks = [pf.Task('t_{}'.format(i)) for i in range(10)]
    pf.Task('done', triggers=pf.all_complete(tasks))
    pf.sequence(tasks)

s

suite sequences
  defstatus suspended
  edit ECF_FILES '/path/to/scratch/files/sequences'
  edit ECF_HOME '/path/to/scratch/out'
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "localhost"
  task t_0
  task t_1
    trigger t_0 eq complete
  task t_2
    trigger t_1 eq complete
  task t_3
    trigger t_2 eq complete
  task t_4
    trigger t_3 eq complete
  task t_5
    trigger t_4 eq complete
  task t_6
    trigger t_5 eq complete
  task t_7
    trigger t_6 eq complete
  task t_8
    trigger t_7 eq complete
  task t_9
    trigger t_8 eq complete
  task done
    trigger ((((((((t_0 eq complete and t_1 eq complete) and t_2 eq complete) and t_3 eq complete) and t_4 eq complete) and t_5 eq complete) and t_6 eq complete) and t_7 eq complete) and t_8 eq complete) and t_9 eq complete
endsuite

A common idiom in looping suites is to have two suites that both loop on dates/times, one which runs behind the other. For example the lag family running after the forecast has completed. This idiom can be expressed more clearly by encapsulating its functionality.

with pf.Suite('follow') as s:
    with pf.Family('leader') as leader:
        pf.RepeatDate("YMD", datetime.date(2019, 1, 1), datetime.date(2019, 12, 31))
    with pf.Family('follower') as follower:
        pf.RepeatDate("YMD", datetime.date(2019, 1, 1), datetime.date(2019, 12, 31))
    follower.follow = leader

s

suite follow
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "default"
  family leader
    repeat date YMD 20190101 20191231 1
  endfamily
  family follower
    trigger leader eq complete or follower:YMD lt leader:YMD
    repeat date YMD 20190101 20191231 1
  endfamily
endsuite

This collection of utility functionality is (perpetually) in progress, and will be updated to account for useful idioms as they emerge.