Chaining Flyte entities

Tags: Basic

Flytekit offers a mechanism for chaining Flyte entities using the >> operator. This is particularly valuable when chaining tasks, subworkflows, and launch plans without the need for data flow between the entities.

Note

To clone and run the example code on this page, see the Flytesnacks repo.

Tasks

Let’s establish a sequence where t1() occurs after t0(), and t2() follows t1().

advanced_composition/chain_entities.py

from flytekit import LaunchPlan, task, workflow


@task
def t2():
    print("Running t2")
    return


@task
def t1():
    print("Running t1")
    return


@task
def t0():
    print("Running t0")
    return


# Chaining tasks
@workflow
def chain_tasks_wf():
    t2_promise = t2()
    t1_promise = t1()
    t0_promise = t0()

    t0_promise >> t1_promise
    t1_promise >> t2_promise

Subworkflows

Just like tasks, you can chain subworkflows.

advanced_composition/chain_entities.py

@workflow
def sub_workflow_1():
    t1()


@workflow
def sub_workflow_0():
    t0()


@workflow
def chain_workflows_wf():
    sub_wf1 = sub_workflow_1()
    sub_wf0 = sub_workflow_0()

    sub_wf0 >> sub_wf1

Launch plans

Like subworkflows, you can chain launch plans.

advanced_composition/chain_entities.py

@workflow
def chain_launchplans_wf():
    lp1 = LaunchPlan.get_or_create(sub_workflow_1, "lp1")()
    lp0 = LaunchPlan.get_or_create(sub_workflow_0, "lp0")()

    lp0 >> lp1

To run the provided workflows on the Flyte cluster, use the following commands:

pyflyte run --remote \
  https://raw.githubusercontent.com/flyteorg/flytesnacks/ab2e8e84362c5b06d2eea0d1d6e29ea7fe460608/examples/advanced_composition/advanced_composition/chain_entities.py \
  chain_tasks_wf

pyflyte run --remote \
  https://raw.githubusercontent.com/flyteorg/flytesnacks/ab2e8e84362c5b06d2eea0d1d6e29ea7fe460608/examples/advanced_composition/advanced_composition/chain_entities.py \
  chain_workflows_wf

pyflyte run --remote \
  https://raw.githubusercontent.com/flyteorg/flytesnacks/ab2e8e84362c5b06d2eea0d1d6e29ea7fe460608/examples/advanced_composition/advanced_composition/chain_entities.py \
  chain_launchplans_wf

Note

Chaining tasks, subworkflows, and launch plans is not supported in local environments. Follow the progress of this issue here.