Contents Menu Expand Light mode Dark mode Auto light/dark mode
Flyte
Logo
Flyte
  • Introduction
  • Quickstart guide
  • Getting started with workflow development
    • Installing development tools
    • Creating a Flyte project
    • Flyte project components
    • Running a workflow locally
  • Flyte fundamentals
    • Tasks, workflows and launch plans
    • Registering workflows
    • Running and scheduling workflows
    • Visualizing task input and output
    • Optimizing tasks
    • Extending Flyte
  • Core use cases
    • Data engineering
    • Machine learning
    • Analytics

Examples

  • User Guide
    • Environment Setup
    • Basics
      • Hello, World!
      • Tasks
      • Workflows
      • Launch Plans
      • Imperative Workflows
      • Documenting Workflows
      • Shell Tasks
      • Named Outputs
    • Data Types and IO
      • Flyte File
      • Flyte Directory
      • Structured Dataset
      • Data Class
      • Accessing Attributes
      • PyTorch Type
      • Enum Type
      • Pickle Type
    • Advanced Composition
      • Conditional
      • Chaining Flyte Entities
      • Subworkflows
      • Dynamic Workflows
      • Map Tasks
      • Eager Workflows
      • Decorating Tasks
      • Decorating Workflows
      • Intratask Checkpoints
      • Waiting for External Inputs
    • Customizing Dependencies
      • Building Docker Images without a Dockerfile
      • Using Raw Containers
      • Multiple Container Images in a Single Workflow
    • Development Lifecycle
      • Writing Agents in Python
      • Pulling Private Images
      • Caching
      • Cache Serializing
      • Decks
      • Creating a New Project
      • Running a Task
      • Running a Workflow
      • Running a Launchplan
      • Inspecting Workflow and Task Executions
      • Debugging Workflow and Task Executions
    • Testing Flyte Tasks and Workflows
      • Mock Tasks for Testing
    • Productionize
      • Customizing Task Resources
      • Reference Task
      • Reference Launch Plan
      • Notifications
      • Scheduling Workflows
      • Configuring Logging Links in UI
      • Configuring Flyte to Access GPUs
      • Using Spot/Preemptible Instances
      • Using Secrets in a Task
      • Adding Workflow Labels and Annotations
    • Extending Flyte
      • Writing Custom Flyte Types
      • Pre-built Container Task Plugin
      • User Container Task Plugin
      • Writing Backend Extensions
      • Container Interface
  • Tutorials
    • Diabetes Classification
      • Train and Validate a Diabetes Classification XGBoost Model
    • House Price Regression
      • Predicting House Price in a Region Using XGBoost
      • Predicting House Price in Multiple Regions Using XGBoost and Dynamic Workflows
    • MNIST Classification With PyTorch and W&B
      • Single Node, Single GPU Training
      • Single Node, Multi GPU Training
    • NLP Processing
      • Word Embeddings and Topic Modelling with Gensim
    • Forecasting Rossman Store Sales with Horovod and Spark
      • Data-Parallel Distributed Training Using Horovod on Spark
    • EDA, Feature Engineering, and Modeling With Papermill
      • Flyte Pipeline in One Jupyter Notebook
      • EDA and Feature Engineering in Jupyter Notebook and Modeling in a Flyte Task
      • EDA and Feature Engineering in One Jupyter Notebook and Modeling in the Other
      • Supermarket Regression Notebook
      • Supermarket Regression 1 Notebook
      • Supermarket Regression 2 Notebook
    • Feast Integration
      • Feature Engineering Tasks
      • Flyte Pipeline with Feast
      • How to Trigger the Feast Workflow using FlyteRemote
    • Nucleotide Sequence Querying with BLASTX
      • BLASTX Example
    • Weather Forecasting
  • Integrations
    • SQL
      • Sqlite3
      • SQLAlchemy
    • Great Expectations
      • Task Example
      • Type Example
    • Papermill
      • Jupyter Notebook Tasks
      • Simple Papermill Notebook
    • Pandera
      • Basic Schema Example
      • Validating and Testing Machine Learning Pipelines
    • Modin
      • KNN Classifier
    • Dolt
      • Quickstart
      • Dolt Branches
    • DBT
      • dbt example
    • WhyLogs
      • whylogs Example
    • MLFlow
      • MLflow Example
    • ONNX
      • PyTorch Example
      • ScikitLearn Example
      • TensorFlow Example
    • DuckDB
      • DuckDB Example
    • Kubernetes Pods
      • Pod Example
    • Dask
      • Running a Dask Task
    • Spark
      • Running a Spark Task
      • Converting a Spark DataFrame to a Pandas DataFrame
    • PyTorch Distributed
      • Run PyTorch Distributed
    • TensorFlow Distributed
      • Run Distributed TensorFlow Training
    • MPI
      • Running Distributed Training Using Horovod and MPI
    • Ray
      • Running Ray Tasks
    • AWS Sagemaker Training
      • Built-in Sagemaker Algorithms
      • Custom Sagemaker Algorithms
    • AWS Sagemaker Pytorch
      • Distributed Pytorch on Sagemaker
    • AWS Athena
      • Athena Query
    • AWS Batch
      • AWS Batch
    • FlyIn
      • FlyIn Vscode Decorator
    • Hive
      • Hive Tasks
    • Memory Machine Cloud
      • Memory Machine Cloud
    • Sensor
      • File Sensor
    • Snowflake
      • Quering data in Snowflake
    • Databricks
      • Running Spark on Databricks
    • BigQuery
      • BigQuery Query
    • Airflow Provider
      • FlyteOperator Example

Cluster Deployment

  • Getting Started
  • Deployment Paths
    • Sandbox Deployment
    • Single Cluster Simple Cloud Deployment
    • Single Cluster Production-grade Cloud Deployment
    • Multiple Kubernetes Cluster Deployment
  • Plugin Setup
    • Configure Kubernetes Plugins
    • Configure AWS Plugins
      • AWS Batch
      • Athena Plugin
      • Sagemaker Plugin Setup
    • Configure GCP Plugins
      • Google BigQuery Plugin
    • Configure Web APIs
      • Snowflake Plugin
      • Databricks Plugin
  • Agent Setup
    • Google BigQuery Agent
    • MMCloud Agent
    • Databricks Agent
    • Sensor Agent
  • Configuration
    • Authenticating in Flyte
    • Migrating Your Authentication Config
    • Understanding Authentication
    • Configuring Custom K8s Resources
    • Adding New Customizable Resources
    • Monitoring
    • Notifications
    • Optimizing Performance
    • Cloud Events
  • Configuration Reference
    • Flyte Datacatalog Configuration
    • Flyte Admin Configuration
    • Flyte Propeller Configuration
    • Flyte Scheduler Configuration
  • Security Overview
  • Flyte API Playground: Swagger

API Reference

  • flytekit
    • Overview
      • Model Files
      • Authoring Structure
      • Control Plane: FlyteRemote
      • Execution Time Support
      • Command Line Interfaces and Clients
    • Core Flytekit
      • flytekit.task
      • flytekit.workflow
      • flytekit.kwtypes
      • flytekit.current_context
      • flytekit.ExecutionParameters
      • flytekit.FlyteContext
      • flytekit.map_task
      • flytekit.core.workflow.ImperativeWorkflow
      • flytekit.core.node_creation.create_node
      • flytekit.core.promise.NodeOutput
      • flytekit.FlyteContextManager
      • flytekit.conditional
      • flytekit.TaskMetadata
      • flytekit.Resources
      • flytekit.WorkflowFailurePolicy
      • flytekit.PodTemplate
      • flytekit.dynamic
      • flytekit.approve
      • flytekit.sleep
      • flytekit.wait_for_input
      • flytekit.CronSchedule
      • flytekit.FixedRate
      • flytekit.Email
      • flytekit.PagerDuty
      • flytekit.Slack
      • flytekit.get_reference_entity
      • flytekit.LaunchPlanReference
      • flytekit.TaskReference
      • flytekit.WorkflowReference
      • flytekit.reference_task
      • flytekit.reference_workflow
      • flytekit.reference_launch_plan
      • flytekit.SQLTask
      • flytekit.ContainerTask
      • flytekit.PythonFunctionTask
      • flytekit.PythonInstanceTask
      • flytekit.LaunchPlan
      • flytekit.Secret
      • flytekit.SecurityContext
      • flytekit.AuthRole
      • flytekit.Labels
      • flytekit.Annotations
      • flytekit.WorkflowExecutionPhase
      • flytekit.Blob
      • flytekit.BlobMetadata
      • flytekit.Literal
      • flytekit.Scalar
      • flytekit.LiteralType
      • flytekit.BlobType
      • flytekit.HashMethod
      • flytekit.Description
      • flytekit.Documentation
      • flytekit.SourceCode
    • Configuration
      • flytekit.configuration.Config
      • flytekit.configuration.Image
      • flytekit.configuration.ImageConfig
      • flytekit.configuration.SerializationSettings
      • flytekit.configuration.FastSerializationSettings
      • flytekit.configuration.PlatformConfig
      • flytekit.configuration.StatsConfig
      • flytekit.configuration.SecretsConfig
      • flytekit.configuration.S3Config
      • flytekit.configuration.GCSConfig
      • flytekit.configuration.DataConfig
    • Remote Access
      • flytekit.remote.remote.FlyteRemote
      • flytekit.remote.remote.Options
      • flytekit.remote.entities.FlyteTask
      • flytekit.remote.entities.FlyteWorkflow
      • flytekit.remote.entities.FlyteLaunchPlan
      • flytekit.remote.entities.FlyteNode
      • flytekit.remote.entities.FlyteTaskNode
      • flytekit.remote.entities.FlyteWorkflowNode
      • flytekit.remote.executions.FlyteWorkflowExecution
      • flytekit.remote.executions.FlyteTaskExecution
      • flytekit.remote.executions.FlyteNodeExecution
    • Clients
      • flytekit.clients.friendly.SynchronousFlyteClient
      • flytekit.clients.raw.RawSynchronousFlyteClient
    • Unit Testing
      • flytekit.testing.patch
      • flytekit.testing.task_mock
    • Specifying Accelerators
    • Extending Flytekit
      • flytekit.extend.get_serializable
      • flytekit.extend.context_manager
      • flytekit.extend.IgnoreOutputs
      • flytekit.extend.ExecutionState
      • flytekit.extend.Image
      • flytekit.extend.ImageConfig
      • flytekit.extend.Interface
      • flytekit.extend.Promise
      • flytekit.extend.TaskPlugins
      • flytekit.extend.DictTransformer
      • flytekit.extend.T
      • flytekit.extend.TypeEngine
      • flytekit.extend.TypeTransformer
      • flytekit.extend.PythonCustomizedContainerTask
      • flytekit.extend.ExecutableTemplateShimTask
      • flytekit.extend.ShimTaskExecutor
    • Flyte Deck
      • flytekit.deck.Deck
      • flytekit.deck.TopFrameRenderer
    • Plugin API reference
      • AWS Athena
        • flytekitplugins.athena.AthenaConfig
        • flytekitplugins.athena.AthenaTask
      • AWS Batch
        • flytekitplugins.awsbatch.AWSBatchConfig
      • Google Bigquery
        • flytekitplugins.bigquery.BigQueryConfig
        • flytekitplugins.bigquery.BigQueryTask
        • flytekitplugins.bigquery.BigQueryAgent
      • Dask
        • flytekitplugins.dask.Dask
        • flytekitplugins.dask.Scheduler
        • flytekitplugins.dask.WorkerGroup
      • Deck standard
        • flytekitplugins.deck.BoxRenderer
        • flytekitplugins.deck.FrameProfilingRenderer
        • flytekitplugins.deck.MarkdownRenderer
        • flytekitplugins.deck.ImageRenderer
        • flytekitplugins.deck.TableRenderer
      • Dolt standard
        • flytekitplugins.dolt.DoltConfig
        • flytekitplugins.dolt.DoltTable
        • flytekitplugins.dolt.DoltTableNameTransformer
      • Great expectations
        • flytekitplugins.great_expectations.BatchRequestConfig
        • flytekitplugins.great_expectations.GreatExpectationsFlyteConfig
        • flytekitplugins.great_expectations.GreatExpectationsTask
        • flytekitplugins.great_expectations.GreatExpectationsType
      • Hive
        • flytekitplugins.hive.HiveConfig
        • flytekitplugins.hive.HiveSelectTask
        • flytekitplugins.hive.HiveTask
      • Pod
        • flytekitplugins.pod.Pod
      • KF MPI
        • flytekitplugins.kfmpi.MPIJob
      • KF Pytorch
        • flytekitplugins.kfpytorch.PyTorch
        • flytekitplugins.kfpytorch.Elastic
      • KF Tensorflow
        • flytekitplugins.kftensorflow.TfJob
      • Modin
        • flytekitplugins.modin.ModinPandasDataFrameTransformer
      • Pandera
        • flytekitplugins.pandera.PanderaTransformer
      • Papermill
        • flytekitplugins.papermill.NotebookTask
        • flytekitplugins.papermill.record_outputs
      • Snowflake
        • flytekitplugins.snowflake.SnowflakeConfig
        • flytekitplugins.snowflake.SnowflakeTask
        • flytekitplugins.snowflake.SnowflakeAgent
      • Spark
        • flytekitplugins.spark.new_spark_session
        • flytekitplugins.spark.ParquetToSparkDecodingHandler
        • flytekitplugins.spark.Spark
        • flytekitplugins.spark.SparkDataFrameSchemaReader
        • flytekitplugins.spark.SparkDataFrameSchemaWriter
        • flytekitplugins.spark.SparkDataFrameTransformer
      • SQLAlchemy
        • flytekitplugins.sqlalchemy.SQLAlchemyConfig
        • flytekitplugins.sqlalchemy.SQLAlchemyDefaultImages
        • flytekitplugins.sqlalchemy.SQLAlchemyTask
      • ONNX PyTorch
        • flytekitplugins.onnxpytorch.PyTorch2ONNXConfig
        • flytekitplugins.onnxpytorch.PyTorch2ONNX
      • ONNX TensorFlow
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNXConfig
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNX
      • ONNX ScikitLearn
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNXConfig
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNX
      • Ray
        • flytekitplugins.ray.HeadNodeConfig
        • flytekitplugins.ray.RayJobConfig
        • flytekitplugins.ray.WorkerNodeConfig
      • DBT
        • flytekitplugins.dbt.DBTRun
        • flytekitplugins.dbt.DBTTest
        • flytekitplugins.dbt.BaseDBTInput
        • flytekitplugins.dbt.BaseDBTOutput
        • flytekitplugins.dbt.DBTRunInput
        • flytekitplugins.dbt.DBTRunOutput
        • flytekitplugins.dbt.DBTTestInput
        • flytekitplugins.dbt.DBTTestOutput
      • Vaex
        • flytekitplugins.vaex.VaexDataFrameToParquetEncodingHandler
        • flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler
      • MLflow
        • flytekitplugins.mlflow.mlflow_autolog
      • DuckDB
        • flytekitplugins.duckdb.DuckDBQuery
    • Custom Tasks
      • flytekit.core.base_task.kwtypes
      • flytekit.core.base_task.PythonTask
      • flytekit.core.base_task.Task
      • flytekit.core.base_task.TaskResolverMixin
      • flytekit.core.base_task.IgnoreOutputs
      • flytekit.core.python_function_task.PythonFunctionTask
      • flytekit.core.python_function_task.PythonInstanceTask
      • Shell Task
        • flytekit.extras.tasks.shell.ShellTask
        • flytekit.extras.tasks.shell.OutputLocation
      • SQLite3 Task
        • flytekit.extras.sqlite3.task.SQLite3Task
        • flytekit.extras.sqlite3.task.SQLite3Config
    • Custom Types
      • Flytekit StructuredDataset
        • flytekit.types.structured.StructuredDataset
        • flytekit.types.structured.StructuredDatasetEncoder
        • flytekit.types.structured.StructuredDatasetDecoder
      • Flytekit File Type
        • flytekit.types.file.FlyteFile
        • flytekit.types.file.HDF5EncodedFile
        • flytekit.types.file.HTMLPage
        • flytekit.types.file.JoblibSerializedFile
        • flytekit.types.file.JPEGImageFile
        • flytekit.types.file.PDFFile
        • flytekit.types.file.PNGImageFile
        • flytekit.types.file.PythonPickledFile
        • flytekit.types.file.PythonNotebook
        • flytekit.types.file.SVGImageFile
      • Flytekit Directory Type
        • flytekit.types.directory.FlyteDirectory
        • flytekit.types.directory.TensorboardLogs
        • flytekit.types.directory.TFRecordsDirectory
      • PyTorch Type
        • flytekit.extras.pytorch.PyTorchCheckpoint
        • flytekit.extras.pytorch.PyTorchCheckpointTransformer
        • flytekit.extras.pytorch.PyTorchModuleTransformer
        • flytekit.extras.pytorch.PyTorchTensorTransformer
      • TensorFlow Type
        • flytekit.extras.tensorflow.TensorFlowRecordFileTransformer
        • flytekit.extras.tensorflow.TensorFlowRecordsDirTransformer
      • Sklearn Type
        • flytekit.extras.sklearn.SklearnEstimatorTransformer
    • Experimental Features
      • flytekit.experimental.map_task
      • flytekit.experimental.eager
      • flytekit.experimental.EagerException
    • Pyflyte CLI
    • Flytekit Contribution Guide
  • flytectl
    • Overview
    • CLI Entrypoint
    • Verbs
      • flytectl create
      • flytectl completion
      • flytectl get
      • flytectl update
      • flytectl delete
      • flytectl register
      • flytectl config
      • flytectl compile
      • flytectl sandbox
      • flytectl demo
      • flytectl version
      • flytectl upgrade
    • Nouns
      • Project
        • flytectl create project
        • flytectl get project
        • flytectl update project
      • Execution
        • flytectl create execution
        • flytectl get execution
        • flytectl update execution
        • flytectl delete execution
      • Workflow
        • flytectl get workflow
        • flytectl update workflow-meta
      • Task
        • flytectl get task
        • flytectl update task-meta
      • Task resource attribute
        • flytectl get task-resource-attribute
        • flytectl update task-resource-attribute
        • flytectl delete task-resource-attribute
      • Cluster resource attribute
        • flytectl get cluster-resource-attribute
        • flytectl delete cluster-resource-attribute
        • flytectl update cluster-resource-attribute
      • Execution cluster label
        • flytectl get execution-cluster-label
        • flytectl update execution-cluster-label
        • flytectl delete execution-cluster-label
      • Execution queue attribute
        • flytectl get execution-queue-attribute
        • flytectl delete execution-queue-attribute
        • flytectl update execution-queue-attribute
      • Plugin override
        • flytectl get plugin-override
        • flytectl update plugin-override
        • flytectl delete plugin-override
      • Launchplan
        • flytectl get launchplan
        • flytectl update launchplan
        • flytectl update launchplan-meta
      • Workflow execution config
        • flytectl get workflow-execution-config
        • flytectl update workflow-execution-config
        • flytectl delete workflow-execution-config
      • Examples
        • flytectl register examples
      • Files
        • flytectl register files
      • Config
        • flytectl config validate
        • flytectl config init
        • flytectl config docs
        • flytectl config discover
      • Sandbox
        • flytectl sandbox start
        • flytectl sandbox status
        • flytectl sandbox teardown
        • flytectl sandbox exec
      • Demo
        • flytectl demo start
        • flytectl demo status
        • flytectl demo teardown
        • flytectl demo exec
        • flytectl demo reload
    • Contributing Guide
  • flyteidl
    • Flyte Core Language Specification
    • FlyteAdmin Service
    • FlyteAdmin REST and gRPC Interface
    • Data Catalog Service
    • Internal and External Eventing Interface
    • Flyte Task Plugin Specification
    • Contributing Guide

Ecosystem

  • flytekit-java
  • unionml
  • pterodactyl
  • latch sdk

Community

  • Community Resources
  • Contributing to Flyte
  • Contributing Examples
  • Roadmap
  • Frequently Asked Questions
  • Troubleshooting Guide

Glossary

  • Main Concepts
    • Tasks
    • Workflows
    • Nodes
    • Launch plans
    • Schedules
    • Registration
    • Executions
    • Understanding the State Transition in a Workflow
    • Timeline of a workflow execution
    • Understand How Flyte Handles Data
    • How to Use Flyte UI
    • What is Data Catalog?
    • Versions
    • Understand the Lifecycle of a Flyte Workflow
  • Control Plane
    • Projects
    • Domains
    • FlyteAdmin
    • FlyteConsole
    • Dynamic Job Spec
  • Component Architecture
    • FlytePropeller Architecture
    • Flyte Native Scheduler Architecture
Flyte
Back to top
Edit this page

Productionize#

In this section, you will learn how to take Flyte pipelines into production. You will explore concepts such as customizing resources, notifications, scheduling, GPU configuration, secrets, spot instances and more.

Customizing Task Resources

Reference Task

Reference Launch Plan

Notifications

Scheduling Workflows

Configuring Logging Links in UI

Configuring Flyte to Access GPUs

Using Spot/Preemptible Instances

Using Secrets in a Task

Adding Workflow Labels and Annotations
Next
Customizing Task Resources
Previous
Mock Tasks for Testing
Copyright © 2022, Flyte Authors
flytekit: dev, 9c9c1ae
flytesnacks: dev, 4ee4c71
flytectl: dev, 9c87165