Deployable Resources

There are many hacks to deploy resources in suites, or resources can be managed and deployed out of band with the suite. It is, however, better to manage versioning of deployed resources in conjunction with the suite. This ensures that a deployed suite always runs what is expected.

pyflow provides a new mechanism for deploying resources. This can include static data files and anything else that should already be in place for tasks to run correctly.

Important

Do not use Resources to deploy scripts or other executable code.

The Resource mechanism provides a decoupling between:

  1. Specifying what resource should be deployed (at suite generation time)

  2. Obtaining the resource, and on which host this resource should be obtained

  3. On which host(s) the resource should be deployed

The host that runs the resource task can be selected by setting the host attribute on the Resources family. The hosts onto which the resources are deployed are specified in a list - this enables the suite to retrieve an external resource only once, even if it needs to be deployed to multiple locations.

[2]:

with pf.Suite('s'):
    with pf.Resources(host=pf.LocalHost()):
        pf.DataResource('script', [pf.LocalHost(resources_directory='/path/to/data'), pf.SSHHost('remote')], 'some data'.encode('utf-8'))

Data can be retrieved from a number of types of location.

[3]:

with pf.Suite('s') as s:
    with pf.Resources():

        # Deploy data directly from the python code
        pf.DataResource('data1', [pf.LocalHost(resources_directory='/path/to/data')], "this is some data".encode('utf-8'))

        # Deploy data from a file accessible at generation time
        pf.FileResource('data2', [pf.LocalHost(resources_directory='/path/to/data')], 'path/to/data.dat')

        # Deploy data accessible from a URL
        pf.WebResource('data3', [pf.LocalHost(resources_directory='/path/to/data')], 'htts://example.com/data')
        pf.WebResource('data4', [pf.LocalHost(resources_directory='/path/to/data')], 'htts://example.com/data', md5='0123456789abcdef')

The resource class can be derived from to obtain more complex resources. The FDB test suite has a MARSResource that runs on a host that has a MARS client to obtain test data from MARS, and which is then transferred to the relevant hosts for testing the FDB tools (which do not have a working and configured MARS client able to interact with the operational MARS).

To extend the functionality of a Resource class, the get_resource member function should be overriden to return an array of lines that can be combined into a script to be run on the Resource execution host to obtain the data.