Suite Basics

Suite

A Suite is a collection of interrelated Tasks. In pyflow suites are described by the pf.Suite class.

[2]:

with pf.Suite('test') as s:
    pf.Task('t1')
    pf.Task('t2')

s

[2]:

suite test
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "default"
  task t1
  task t2
endsuite

The suite definition describes how your tasks run and interact.

Task

A Task is a building block of a suite and encapsulates a single job to be executed. In pyflow tasks are described by the pf.Task class.

[3]:

with pf.Suite('test') as s:
    pf.Task('t1', script='whoami', labels={'a_label': 'with a value'}, defstatus=pf.state.suspended)

s

[3]:

suite test
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "default"
  task t1
    defstatus suspended
    label a_label "with a value"
endsuite

Each task can have a corresponding Script which does the actual heavy lifting.

[4]:

s.deploy_suite(pf.Notebook)

[4]:

File: None

#!/bin/bash

echo "Running on: $(hostname)" || true
set -uex


export ECF_PORT=%ECF_PORT%    # The server port number
export ECF_HOST=%ECF_HOST%    # The host name where the server is running
export ECF_NAME=%ECF_NAME%    # The name of this current task
export ECF_PASS=%ECF_PASS%    # A unique password
export ECF_TRYNO=%ECF_TRYNO%  # Current try number of the task

echo "Current working directory: $(pwd)"

%nopp

whoami

%end

Family

Tasks can be logically grouped together in Families, which themselves may be placed in other families and/or suites. A family is described by the pf.Family class. All the entities (tasks, families and suites) are called Nodes and form a hierarchical tree.

[5]:

with pf.Suite('test') as s:
    with pf.Family('f1'):
        pf.Task('t1')
        pf.Task('t2')

s

[5]:

suite test
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "default"
  family f1
    task t1
    task t2
  endfamily
endsuite

You can picture a suite as a hierarchical structure very similar to a UNIX file system, where the families are the directories, and the tasks are the files.

The suite is a family with some extra attributes, like dates and clocks. Like directories, families can themselves contain other families. And like directories, there can be many tasks with the same name, as long as they are in different families.

Unless you tell pyflow where to find specific files, the default behaviour is to expect the file structure to reflect the structure of the suite.

Scripts

Scripts are text files with an .ecf extension that correspond to the task in the suite definition. The script defines the main work that is to be carried out. The script includes child commands, special comments, and manual sections that provide information for users.

%include <head.h>
echo "I am part of a suite that lives in %ECF_HOME%"
%include <tail.h>

The child commands are a restricted set of ecflow_client commands that communicate with the ecFlow server. They inform the server when an event occurs, for example a label is set.

ecflow_client --label=info "I have now finished my work."

Hosts

To define an execution contexts to suites, pyflow provides the pf.Host class. It is required to define a host object for a suite in order to generate the executable nodes. The host can be set at any level (Suite, Family or Task) and is inherited unless overridden.

[6]:

with pf.Suite('test', host=pf.LocalHost(), files='/test') as s:
    with pf.Family('f1'):
        pf.Task('t1', script=[
            'cmake -DCMAKE_INSTALL_PREFIX=/usr/local .',
            'cmake --build .',
            'ctest --output-on-failure .',
            'cmake --install .',
        ])

s

[6]:

suite test
  edit ECF_FILES '/test'
  edit ECF_JOB_CMD 'bash -c 'export ECF_PORT=%ECF_PORT%; export ECF_HOST=%ECF_HOST%; export ECF_NAME=%ECF_NAME%; export ECF_PASS=%ECF_PASS%; export ECF_TRYNO=%ECF_TRYNO%; export PATH=/usr/local/apps/ecflow/%ECF_VERSION%/bin:$PATH; ecflow_client --init="$$" && %ECF_JOB% && ecflow_client --complete || ecflow_client --abort ' 1> %ECF_JOBOUT% 2>&1 &'
  edit ECF_KILL_CMD 'pkill -15 -P %ECF_RID%'
  edit ECF_STATUS_CMD 'true'
  edit ECF_OUT '%ECF_HOME%'
  label exec_host "localhost"
  family f1
    task t1
  endfamily
endsuite

[7]:

s.deploy_suite(pf.Notebook)

[7]:

File: /test/t1.ecf

#!/bin/bash

echo "Running on: $(hostname)" || true
set -uex


export ECF_PORT=%ECF_PORT%    # The server port number
export ECF_HOST=%ECF_HOST%    # The host name where the server is running
export ECF_NAME=%ECF_NAME%    # The name of this current task
export ECF_PASS=%ECF_PASS%    # A unique password
export ECF_TRYNO=%ECF_TRYNO%  # Current try number of the task

echo "Current working directory: $(pwd)"

%nopp

cmake -DCMAKE_INSTALL_PREFIX=/usr/local .
cmake --build .
ctest --output-on-failure .
cmake --install .

%end