Contents Menu Expand Light mode Dark mode Auto light/dark mode
Flyte
Logo
Flyte
  • Introduction
  • Quickstart guide
  • Getting started with workflow development
    • Installing development tools
    • Creating a Flyte project
    • Flyte project components
    • Running a workflow locally
  • Flyte fundamentals
    • Tasks, workflows and launch plans
    • Registering workflows
    • Running and scheduling workflows
    • Visualizing task input and output
    • Optimizing tasks
    • Extending Flyte
  • Core use cases
    • Data engineering
    • Machine learning
    • Analytics

Examples

  • User Guide
    • Environment Setup
    • Basics
      • Hello, World!
      • Tasks
      • Workflows
      • Launch Plans
      • Imperative Workflows
      • Documenting Workflows
      • Shell Tasks
      • Named Outputs
    • Data Types and IO
      • Flyte File
      • Flyte Directory
      • Structured Dataset
      • Data Class
      • Accessing Attributes
      • PyTorch Type
      • Enum Type
      • Pickle Type
    • Advanced Composition
      • Conditional
      • Chaining Flyte Entities
      • Subworkflows
      • Dynamic Workflows
      • Map Tasks
      • Eager Workflows
      • Decorating Tasks
      • Decorating Workflows
      • Intratask Checkpoints
      • Waiting for External Inputs
    • Customizing Dependencies
      • Building Docker Images without a Dockerfile
      • Using Raw Containers
      • Multiple Container Images in a Single Workflow
    • Development Lifecycle
      • Writing Agents in Python
      • Pulling Private Images
      • Caching
      • Cache Serializing
      • Decks
      • Creating a New Project
      • Running a Task
      • Running a Workflow
      • Running a Launchplan
      • Inspecting Workflow and Task Executions
      • Debugging Workflow and Task Executions
    • Testing Flyte Tasks and Workflows
      • Mock Tasks for Testing
    • Productionize
      • Customizing Task Resources
      • Reference Task
      • Reference Launch Plan
      • Notifications
      • Scheduling Workflows
      • Configuring Logging Links in UI
      • Configuring Flyte to Access GPUs
      • Using Spot/Preemptible Instances
      • Using Secrets in a Task
      • Adding Workflow Labels and Annotations
    • Extending Flyte
      • Writing Custom Flyte Types
      • Pre-built Container Task Plugin
      • User Container Task Plugin
      • Writing Backend Extensions
      • Container Interface
  • Tutorials
    • Diabetes Classification
      • Train and Validate a Diabetes Classification XGBoost Model
    • House Price Regression
      • Predicting House Price in a Region Using XGBoost
      • Predicting House Price in Multiple Regions Using XGBoost and Dynamic Workflows
    • MNIST Classification With PyTorch and W&B
      • Single Node, Single GPU Training
      • Single Node, Multi GPU Training
    • NLP Processing
      • Word Embeddings and Topic Modelling with Gensim
    • Forecasting Rossman Store Sales with Horovod and Spark
      • Data-Parallel Distributed Training Using Horovod on Spark
    • EDA, Feature Engineering, and Modeling With Papermill
      • Flyte Pipeline in One Jupyter Notebook
      • EDA and Feature Engineering in Jupyter Notebook and Modeling in a Flyte Task
      • EDA and Feature Engineering in One Jupyter Notebook and Modeling in the Other
      • Supermarket Regression Notebook
      • Supermarket Regression 1 Notebook
      • Supermarket Regression 2 Notebook
    • Feast Integration
      • Feature Engineering Tasks
      • Flyte Pipeline with Feast
      • How to Trigger the Feast Workflow using FlyteRemote
    • Nucleotide Sequence Querying with BLASTX
      • BLASTX Example
    • Weather Forecasting
  • Integrations
    • SQL
      • Sqlite3
      • SQLAlchemy
    • Great Expectations
      • Task Example
      • Type Example
    • Papermill
      • Jupyter Notebook Tasks
      • Simple Papermill Notebook
    • Pandera
      • Basic Schema Example
      • Validating and Testing Machine Learning Pipelines
    • Modin
      • KNN Classifier
    • Dolt
      • Quickstart
      • Dolt Branches
    • DBT
      • dbt example
    • WhyLogs
      • whylogs Example
    • MLFlow
      • MLflow Example
    • ONNX
      • PyTorch Example
      • ScikitLearn Example
      • TensorFlow Example
    • DuckDB
      • DuckDB Example
    • Kubernetes Pods
      • Pod Example
    • Dask
      • Running a Dask Task
    • Spark
      • Running a Spark Task
      • Converting a Spark DataFrame to a Pandas DataFrame
    • PyTorch Distributed
      • Run PyTorch Distributed
    • TensorFlow Distributed
      • Run Distributed TensorFlow Training
    • MPI
      • Running Distributed Training Using Horovod and MPI
    • Ray
      • Running Ray Tasks
    • AWS Sagemaker Training
      • Built-in Sagemaker Algorithms
      • Custom Sagemaker Algorithms
    • AWS Sagemaker Pytorch
      • Distributed Pytorch on Sagemaker
    • AWS Athena
      • Athena Query
    • AWS Batch
      • AWS Batch
    • FlyIn
      • FlyIn Vscode Decorator
    • Hive
      • Hive Tasks
    • Memory Machine Cloud
      • Memory Machine Cloud
    • Sensor
      • File Sensor
    • Snowflake
      • Quering data in Snowflake
    • Databricks
      • Running Spark on Databricks
    • BigQuery
      • BigQuery Query
    • Airflow Provider
      • FlyteOperator Example

Cluster Deployment

  • Getting Started
  • Deployment Paths
    • Sandbox Deployment
    • Single Cluster Simple Cloud Deployment
    • Single Cluster Production-grade Cloud Deployment
    • Multiple Kubernetes Cluster Deployment
  • Plugin Setup
    • Configure Kubernetes Plugins
    • Configure AWS Plugins
      • AWS Batch
      • Athena Plugin
      • Sagemaker Plugin Setup
    • Configure GCP Plugins
      • Google BigQuery Plugin
    • Configure Web APIs
      • Snowflake Plugin
      • Databricks Plugin
  • Agent Setup
    • Google BigQuery Agent
    • MMCloud Agent
    • Databricks Agent
    • Sensor Agent
  • Configuration
    • Authenticating in Flyte
    • Migrating Your Authentication Config
    • Understanding Authentication
    • Configuring Custom K8s Resources
    • Adding New Customizable Resources
    • Monitoring
    • Notifications
    • Optimizing Performance
    • Cloud Events
  • Configuration Reference
    • Flyte Datacatalog Configuration
    • Flyte Admin Configuration
    • Flyte Propeller Configuration
    • Flyte Scheduler Configuration
  • Security Overview
  • Flyte API Playground: Swagger

API Reference

  • flytekit
    • Overview
      • Model Files
      • Authoring Structure
      • Control Plane: FlyteRemote
      • Execution Time Support
      • Command Line Interfaces and Clients
    • Core Flytekit
      • flytekit.task
      • flytekit.workflow
      • flytekit.kwtypes
      • flytekit.current_context
      • flytekit.ExecutionParameters
      • flytekit.FlyteContext
      • flytekit.map_task
      • flytekit.core.workflow.ImperativeWorkflow
      • flytekit.core.node_creation.create_node
      • flytekit.core.promise.NodeOutput
      • flytekit.FlyteContextManager
      • flytekit.conditional
      • flytekit.TaskMetadata
      • flytekit.Resources
      • flytekit.WorkflowFailurePolicy
      • flytekit.PodTemplate
      • flytekit.dynamic
      • flytekit.approve
      • flytekit.sleep
      • flytekit.wait_for_input
      • flytekit.CronSchedule
      • flytekit.FixedRate
      • flytekit.Email
      • flytekit.PagerDuty
      • flytekit.Slack
      • flytekit.get_reference_entity
      • flytekit.LaunchPlanReference
      • flytekit.TaskReference
      • flytekit.WorkflowReference
      • flytekit.reference_task
      • flytekit.reference_workflow
      • flytekit.reference_launch_plan
      • flytekit.SQLTask
      • flytekit.ContainerTask
      • flytekit.PythonFunctionTask
      • flytekit.PythonInstanceTask
      • flytekit.LaunchPlan
      • flytekit.Secret
      • flytekit.SecurityContext
      • flytekit.AuthRole
      • flytekit.Labels
      • flytekit.Annotations
      • flytekit.WorkflowExecutionPhase
      • flytekit.Blob
      • flytekit.BlobMetadata
      • flytekit.Literal
      • flytekit.Scalar
      • flytekit.LiteralType
      • flytekit.BlobType
      • flytekit.HashMethod
      • flytekit.Description
      • flytekit.Documentation
      • flytekit.SourceCode
    • Configuration
      • flytekit.configuration.Config
      • flytekit.configuration.Image
      • flytekit.configuration.ImageConfig
      • flytekit.configuration.SerializationSettings
      • flytekit.configuration.FastSerializationSettings
      • flytekit.configuration.PlatformConfig
      • flytekit.configuration.StatsConfig
      • flytekit.configuration.SecretsConfig
      • flytekit.configuration.S3Config
      • flytekit.configuration.GCSConfig
      • flytekit.configuration.DataConfig
    • Remote Access
      • flytekit.remote.remote.FlyteRemote
      • flytekit.remote.remote.Options
      • flytekit.remote.entities.FlyteTask
      • flytekit.remote.entities.FlyteWorkflow
      • flytekit.remote.entities.FlyteLaunchPlan
      • flytekit.remote.entities.FlyteNode
      • flytekit.remote.entities.FlyteTaskNode
      • flytekit.remote.entities.FlyteWorkflowNode
      • flytekit.remote.executions.FlyteWorkflowExecution
      • flytekit.remote.executions.FlyteTaskExecution
      • flytekit.remote.executions.FlyteNodeExecution
    • Clients
      • flytekit.clients.friendly.SynchronousFlyteClient
      • flytekit.clients.raw.RawSynchronousFlyteClient
    • Unit Testing
      • flytekit.testing.patch
      • flytekit.testing.task_mock
    • Specifying Accelerators
    • Extending Flytekit
      • flytekit.extend.get_serializable
      • flytekit.extend.context_manager
      • flytekit.extend.IgnoreOutputs
      • flytekit.extend.ExecutionState
      • flytekit.extend.Image
      • flytekit.extend.ImageConfig
      • flytekit.extend.Interface
      • flytekit.extend.Promise
      • flytekit.extend.TaskPlugins
      • flytekit.extend.DictTransformer
      • flytekit.extend.T
      • flytekit.extend.TypeEngine
      • flytekit.extend.TypeTransformer
      • flytekit.extend.PythonCustomizedContainerTask
      • flytekit.extend.ExecutableTemplateShimTask
      • flytekit.extend.ShimTaskExecutor
    • Flyte Deck
      • flytekit.deck.Deck
      • flytekit.deck.TopFrameRenderer
    • Plugin API reference
      • AWS Athena
        • flytekitplugins.athena.AthenaConfig
        • flytekitplugins.athena.AthenaTask
      • AWS Batch
        • flytekitplugins.awsbatch.AWSBatchConfig
      • Google Bigquery
        • flytekitplugins.bigquery.BigQueryConfig
        • flytekitplugins.bigquery.BigQueryTask
        • flytekitplugins.bigquery.BigQueryAgent
      • Dask
        • flytekitplugins.dask.Dask
        • flytekitplugins.dask.Scheduler
        • flytekitplugins.dask.WorkerGroup
      • Deck standard
        • flytekitplugins.deck.BoxRenderer
        • flytekitplugins.deck.FrameProfilingRenderer
        • flytekitplugins.deck.MarkdownRenderer
        • flytekitplugins.deck.ImageRenderer
        • flytekitplugins.deck.TableRenderer
      • Dolt standard
        • flytekitplugins.dolt.DoltConfig
        • flytekitplugins.dolt.DoltTable
        • flytekitplugins.dolt.DoltTableNameTransformer
      • Great expectations
        • flytekitplugins.great_expectations.BatchRequestConfig
        • flytekitplugins.great_expectations.GreatExpectationsFlyteConfig
        • flytekitplugins.great_expectations.GreatExpectationsTask
        • flytekitplugins.great_expectations.GreatExpectationsType
      • Hive
        • flytekitplugins.hive.HiveConfig
        • flytekitplugins.hive.HiveSelectTask
        • flytekitplugins.hive.HiveTask
      • Pod
        • flytekitplugins.pod.Pod
      • KF MPI
        • flytekitplugins.kfmpi.MPIJob
      • KF Pytorch
        • flytekitplugins.kfpytorch.PyTorch
        • flytekitplugins.kfpytorch.Elastic
      • KF Tensorflow
        • flytekitplugins.kftensorflow.TfJob
      • Modin
        • flytekitplugins.modin.ModinPandasDataFrameTransformer
      • Pandera
        • flytekitplugins.pandera.PanderaTransformer
      • Papermill
        • flytekitplugins.papermill.NotebookTask
        • flytekitplugins.papermill.record_outputs
      • Snowflake
        • flytekitplugins.snowflake.SnowflakeConfig
        • flytekitplugins.snowflake.SnowflakeTask
        • flytekitplugins.snowflake.SnowflakeAgent
      • Spark
        • flytekitplugins.spark.new_spark_session
        • flytekitplugins.spark.ParquetToSparkDecodingHandler
        • flytekitplugins.spark.Spark
        • flytekitplugins.spark.SparkDataFrameSchemaReader
        • flytekitplugins.spark.SparkDataFrameSchemaWriter
        • flytekitplugins.spark.SparkDataFrameTransformer
      • SQLAlchemy
        • flytekitplugins.sqlalchemy.SQLAlchemyConfig
        • flytekitplugins.sqlalchemy.SQLAlchemyDefaultImages
        • flytekitplugins.sqlalchemy.SQLAlchemyTask
      • ONNX PyTorch
        • flytekitplugins.onnxpytorch.PyTorch2ONNXConfig
        • flytekitplugins.onnxpytorch.PyTorch2ONNX
      • ONNX TensorFlow
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNXConfig
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNX
      • ONNX ScikitLearn
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNXConfig
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNX
      • Ray
        • flytekitplugins.ray.HeadNodeConfig
        • flytekitplugins.ray.RayJobConfig
        • flytekitplugins.ray.WorkerNodeConfig
      • DBT
        • flytekitplugins.dbt.DBTRun
        • flytekitplugins.dbt.DBTTest
        • flytekitplugins.dbt.BaseDBTInput
        • flytekitplugins.dbt.BaseDBTOutput
        • flytekitplugins.dbt.DBTRunInput
        • flytekitplugins.dbt.DBTRunOutput
        • flytekitplugins.dbt.DBTTestInput
        • flytekitplugins.dbt.DBTTestOutput
      • Vaex
        • flytekitplugins.vaex.VaexDataFrameToParquetEncodingHandler
        • flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler
      • MLflow
        • flytekitplugins.mlflow.mlflow_autolog
      • DuckDB
        • flytekitplugins.duckdb.DuckDBQuery
    • Custom Tasks
      • flytekit.core.base_task.kwtypes
      • flytekit.core.base_task.PythonTask
      • flytekit.core.base_task.Task
      • flytekit.core.base_task.TaskResolverMixin
      • flytekit.core.base_task.IgnoreOutputs
      • flytekit.core.python_function_task.PythonFunctionTask
      • flytekit.core.python_function_task.PythonInstanceTask
      • Shell Task
        • flytekit.extras.tasks.shell.ShellTask
        • flytekit.extras.tasks.shell.OutputLocation
      • SQLite3 Task
        • flytekit.extras.sqlite3.task.SQLite3Task
        • flytekit.extras.sqlite3.task.SQLite3Config
    • Custom Types
      • Flytekit StructuredDataset
        • flytekit.types.structured.StructuredDataset
        • flytekit.types.structured.StructuredDatasetEncoder
        • flytekit.types.structured.StructuredDatasetDecoder
      • Flytekit File Type
        • flytekit.types.file.FlyteFile
        • flytekit.types.file.HDF5EncodedFile
        • flytekit.types.file.HTMLPage
        • flytekit.types.file.JoblibSerializedFile
        • flytekit.types.file.JPEGImageFile
        • flytekit.types.file.PDFFile
        • flytekit.types.file.PNGImageFile
        • flytekit.types.file.PythonPickledFile
        • flytekit.types.file.PythonNotebook
        • flytekit.types.file.SVGImageFile
      • Flytekit Directory Type
        • flytekit.types.directory.FlyteDirectory
        • flytekit.types.directory.TensorboardLogs
        • flytekit.types.directory.TFRecordsDirectory
      • PyTorch Type
        • flytekit.extras.pytorch.PyTorchCheckpoint
        • flytekit.extras.pytorch.PyTorchCheckpointTransformer
        • flytekit.extras.pytorch.PyTorchModuleTransformer
        • flytekit.extras.pytorch.PyTorchTensorTransformer
      • TensorFlow Type
        • flytekit.extras.tensorflow.TensorFlowRecordFileTransformer
        • flytekit.extras.tensorflow.TensorFlowRecordsDirTransformer
      • Sklearn Type
        • flytekit.extras.sklearn.SklearnEstimatorTransformer
    • Experimental Features
      • flytekit.experimental.map_task
      • flytekit.experimental.eager
      • flytekit.experimental.EagerException
    • Pyflyte CLI
    • Flytekit Contribution Guide
  • flytectl
    • Overview
    • CLI Entrypoint
    • Verbs
      • flytectl create
      • flytectl completion
      • flytectl get
      • flytectl update
      • flytectl delete
      • flytectl register
      • flytectl config
      • flytectl compile
      • flytectl sandbox
      • flytectl demo
      • flytectl version
      • flytectl upgrade
    • Nouns
      • Project
        • flytectl create project
        • flytectl get project
        • flytectl update project
      • Execution
        • flytectl create execution
        • flytectl get execution
        • flytectl update execution
        • flytectl delete execution
      • Workflow
        • flytectl get workflow
        • flytectl update workflow-meta
      • Task
        • flytectl get task
        • flytectl update task-meta
      • Task resource attribute
        • flytectl get task-resource-attribute
        • flytectl update task-resource-attribute
        • flytectl delete task-resource-attribute
      • Cluster resource attribute
        • flytectl get cluster-resource-attribute
        • flytectl delete cluster-resource-attribute
        • flytectl update cluster-resource-attribute
      • Execution cluster label
        • flytectl get execution-cluster-label
        • flytectl update execution-cluster-label
        • flytectl delete execution-cluster-label
      • Execution queue attribute
        • flytectl get execution-queue-attribute
        • flytectl delete execution-queue-attribute
        • flytectl update execution-queue-attribute
      • Plugin override
        • flytectl get plugin-override
        • flytectl update plugin-override
        • flytectl delete plugin-override
      • Launchplan
        • flytectl get launchplan
        • flytectl update launchplan
        • flytectl update launchplan-meta
      • Workflow execution config
        • flytectl get workflow-execution-config
        • flytectl update workflow-execution-config
        • flytectl delete workflow-execution-config
      • Examples
        • flytectl register examples
      • Files
        • flytectl register files
      • Config
        • flytectl config validate
        • flytectl config init
        • flytectl config docs
        • flytectl config discover
      • Sandbox
        • flytectl sandbox start
        • flytectl sandbox status
        • flytectl sandbox teardown
        • flytectl sandbox exec
      • Demo
        • flytectl demo start
        • flytectl demo status
        • flytectl demo teardown
        • flytectl demo exec
        • flytectl demo reload
    • Contributing Guide
  • flyteidl
    • Flyte Core Language Specification
    • FlyteAdmin Service
    • FlyteAdmin REST and gRPC Interface
    • Data Catalog Service
    • Internal and External Eventing Interface
    • Flyte Task Plugin Specification
    • Contributing Guide

Ecosystem

  • flytekit-java
  • unionml
  • pterodactyl
  • latch sdk

Community

  • Community Resources
  • Contributing to Flyte
  • Contributing Examples
  • Roadmap
  • Frequently Asked Questions
  • Troubleshooting Guide

Glossary

  • Main Concepts
    • Tasks
    • Workflows
    • Nodes
    • Launch plans
    • Schedules
    • Registration
    • Executions
    • Understanding the State Transition in a Workflow
    • Timeline of a workflow execution
    • Understand How Flyte Handles Data
    • How to Use Flyte UI
    • What is Data Catalog?
    • Versions
    • Understand the Lifecycle of a Flyte Workflow
  • Control Plane
    • Projects
    • Domains
    • FlyteAdmin
    • FlyteConsole
    • Dynamic Job Spec
  • Component Architecture
    • FlytePropeller Architecture
    • Flyte Native Scheduler Architecture
Flyte
Back to top
Edit this page

Tag: Advanced#

With this tag

  • Understanding Authentication
  • Platform Events
  • Cloud Events
  • Notifications
  • Optimizing Performance
  • Authenticating in Flyte
  • Migrating Your Authentication Config
  • Monitoring
  • Adding New Customizable Resources
  • Single Cluster Production-grade Cloud Deployment
  • Multiple Kubernetes Cluster Deployment
  • Security Overview
  • Configure Kubernetes Plugins
  • What is Data Catalog?
  • FlyteAdmin
  • Component Architecture
  • Flyte Native Scheduler Architecture
  • FlytePropeller Architecture
Copyright © 2022, Flyte Authors
flytekit: dev, 9c9c1ae
flytesnacks: dev, 4ee4c71
flytectl: dev, 9c87165