Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode
Flyte
Logo
Flyte
  • Introduction
  • Quickstart guide
  • Getting started with workflow development
    • Installing development tools
    • Creating a Flyte project
    • Flyte project components
    • Running a workflow locally
  • Flyte fundamentals
    • Tasks, workflows and launch plans
    • Registering workflows
    • Running and scheduling workflows
    • Visualizing task input and output
    • Optimizing tasks
    • Extending Flyte
  • Flyte agents
    • Testing agents in a local Python environment
    • Enabling agents in your Flyte deployment
    • Developing agents
    • Testing agents in a local development cluster
    • Deploying agents to the Flyte sandbox
    • Implementing the agent metadata service
  • Core use cases
    • Data engineering
    • Machine learning
    • Analytics

Examples

  • User Guide
    • Environment setup
    • Basics
      • Hello, World!
      • Tasks
      • Workflows
      • Launch plans
      • Imperative workflows
      • Documenting workflows
      • Shell tasks
      • Named outputs
    • Data Types and IO
      • FlyteFile
      • FlyteDirectory
      • StructuredDataset
      • Dataclass
      • Accessing attributes
      • PyTorch type
      • Enum type
      • Pickle type
    • Advanced composition
      • Conditionals
      • Chaining Flyte entities
      • Subworkflows
      • Dynamic workflows
      • Map tasks
      • Nested parallelization
      • Eager workflows
      • Decorating tasks
      • Decorating workflows
      • Intratask checkpoints
      • Waiting for external inputs
    • Customizing dependencies
      • ImageSpec
      • Raw containers
      • Multiple images in a workflow
    • Development lifecycle
      • Private images
      • Caching
      • Cache serializing
      • Decks
      • Failure node
      • Creating a new project
      • Running tasks
      • Running workflows
      • Running launch plans
      • Inspecting executions
      • Debugging executions
      • Migrating from Airflow to Flyte
      • For current Flyte users
    • Testing
      • Mocking tasks
    • Productionize
      • Customizing task resources
      • Reference tasks
      • Reference launch plans
      • Notifications
      • Schedules
      • Configuring logging links in the UI
      • Configuring access to GPUs
      • Spot instances
      • Secrets
      • Workflow labels and annotations
    • Extending Flyte
      • Custom types
      • Prebuilt container task plugins
      • User container task plugins
      • Backend plugins
      • Container interface
  • Tutorials
    • Diabetes Classification
      • Train and Validate a Diabetes Classification XGBoost Model
    • House Price Regression
      • Predicting House Price in a Region Using XGBoost
      • Predicting House Price in Multiple Regions Using XGBoost and Dynamic Workflows
    • MNIST Classification With PyTorch and W&B
      • Single Node, Single GPU Training
      • Single Node, Multi GPU Training
    • NLP Processing
      • Word Embeddings and Topic Modelling with Gensim
    • Forecasting Rossman Store Sales with Horovod and Spark
      • Data-Parallel Distributed Training Using Horovod on Spark
    • EDA, Feature Engineering, and Modeling With Papermill
      • Flyte Pipeline in One Jupyter Notebook
      • EDA and Feature Engineering in Jupyter Notebook and Modeling in a Flyte Task
      • EDA and Feature Engineering in One Jupyter Notebook and Modeling in the Other
      • Supermarket Regression Notebook
      • Supermarket Regression 1 Notebook
      • Supermarket Regression 2 Notebook
    • Feast Integration
      • Feature Engineering Tasks
      • Flyte Pipeline with Feast
      • How to Trigger the Feast Workflow using FlyteRemote
    • Nucleotide Sequence Querying with BLASTX
      • BLASTX Example
    • Weather Forecasting
  • Integrations
    • SQL
      • Sqlite3
      • SQLAlchemy
    • Great Expectations
      • Task Example
      • Type Example
    • Papermill
      • Jupyter Notebook Tasks
      • Simple Papermill Notebook
    • Pandera
      • Basic Schema Example
      • Validating and Testing Machine Learning Pipelines
    • Modin
      • KNN Classifier
    • Dolt
      • Quickstart
      • Dolt Branches
    • DBT
      • dbt example
    • WhyLogs
      • whylogs Example
    • MLFlow
      • MLflow Example
    • ONNX
      • PyTorch Example
      • ScikitLearn Example
      • TensorFlow Example
    • DuckDB
      • DuckDB Example
    • Weights and Biases
      • Weights and Biases Example
    • Kubernetes Pods
      • Pod Example
    • Dask
      • Running a Dask Task
    • Spark
      • Running a Spark Task
      • Converting a Spark DataFrame to a Pandas DataFrame
    • PyTorch Distributed
      • Run PyTorch Distributed
      • Use PyTorch Lightning to Train an MNIST Autoencoder
    • TensorFlow Distributed
      • Run Distributed TensorFlow Training
    • MPI
      • Running Distributed Training Using Horovod and MPI
    • Ray
      • Running Ray Tasks
    • Airflow agent
      • Airflow agent example usage
    • BigQuery agent
      • BigQuery agent example usage
    • ChatGPT agent
      • ChatGPT agent example usage
    • Databricks agent example
      • Running Spark on Databricks
    • Memory Machine Cloud agent
      • Memory Machine Cloud agent example usage
    • OpenAI Batch Agent
      • Batching Requests for Asynchronous Processing
    • AWS SageMaker Inference Agent
      • Deploy and serve an XGBoost model on AWS SageMaker using FastAPI
    • Sensor
      • File Sensor
    • Snowflake agent
      • Querying data in Snowflake
    • AWS Athena
      • Athena Query
    • AWS Batch
      • AWS Batch
    • FlyteInteractive
      • FlyteInteractive Vscode Decorator
      • FlyteInteractive Jupyter Decorator
    • Hive
      • Hive Tasks
    • Airflow Provider
      • FlyteOperator Example
  • Deprecated integrations
    • BigQuery plugin
      • BigQuery example query
    • Databricks plugin
      • Running Spark on Databricks
    • Snowflake plugin
      • Querying data in Snowflake

Cluster Deployment

  • Getting Started
  • Deployment Paths
    • Sandbox Deployment
    • Single Cluster Simple Cloud Deployment
    • Single Cluster Production-grade Cloud Deployment
    • Multiple Kubernetes Cluster Deployment
  • Plugin Setup
    • Configure Kubernetes Plugins
    • Configure AWS Plugins
      • AWS Batch
      • Athena Plugin
      • Sagemaker Plugin Setup
    • Configure GCP Plugins
      • Google BigQuery Plugin
    • Configure Web APIs
      • Snowflake Plugin
      • Databricks Plugin
  • Agent Setup
    • Airflow agent
    • ChatGPT agent
    • Databricks agent
    • Google BigQuery agent
    • MMCloud Agent
    • SageMaker Inference Agent
    • Sensor agent
    • Snowflake agent
    • OpenAI Batch Agent
  • Configuration
    • Authenticating in Flyte
    • Migrating Your Authentication Config
    • Understanding Authentication
    • Configuring task pods with K8s PodTemplates
    • Customizing project, domain, and workflow resources with flytectl
    • Monitoring
    • Notifications
    • Optimizing Performance
    • Cloud Events
    • Flyte ResourceManager
  • Configuration Reference
    • Flyte Datacatalog Configuration
    • Flyte Admin Configuration
    • Flyte Propeller Configuration
    • Flyte Scheduler Configuration
  • Security Overview
  • Flyte API Playground: Swagger

API Reference

  • flytekit
    • Overview
      • Model Files
      • Authoring Structure
      • Control Plane: FlyteRemote
      • Execution Time Support
      • Command Line Interfaces and Clients
    • Core Flytekit
      • flytekit.task
      • flytekit.workflow
      • flytekit.kwtypes
      • flytekit.current_context
      • flytekit.ExecutionParameters
      • flytekit.FlyteContext
      • flytekit.map_task
      • flytekit.core.workflow.ImperativeWorkflow
      • flytekit.core.node_creation.create_node
      • flytekit.core.promise.NodeOutput
      • flytekit.FlyteContextManager
      • flytekit.conditional
      • flytekit.TaskMetadata
      • flytekit.Resources
      • flytekit.WorkflowFailurePolicy
      • flytekit.PodTemplate
      • flytekit.dynamic
      • flytekit.approve
      • flytekit.sleep
      • flytekit.wait_for_input
      • flytekit.CronSchedule
      • flytekit.FixedRate
      • flytekit.Email
      • flytekit.PagerDuty
      • flytekit.Slack
      • flytekit.get_reference_entity
      • flytekit.LaunchPlanReference
      • flytekit.TaskReference
      • flytekit.WorkflowReference
      • flytekit.reference_task
      • flytekit.reference_workflow
      • flytekit.reference_launch_plan
      • flytekit.SQLTask
      • flytekit.ContainerTask
      • flytekit.PythonFunctionTask
      • flytekit.PythonInstanceTask
      • flytekit.LaunchPlan
      • flytekit.Secret
      • flytekit.SecurityContext
      • flytekit.AuthRole
      • flytekit.Labels
      • flytekit.Annotations
      • flytekit.WorkflowExecutionPhase
      • flytekit.Blob
      • flytekit.BlobMetadata
      • flytekit.Literal
      • flytekit.Scalar
      • flytekit.LiteralType
      • flytekit.BlobType
      • flytekit.HashMethod
      • flytekit.Artifact
      • flytekit.Description
      • flytekit.Documentation
      • flytekit.SourceCode
    • Configuration
      • flytekit.configuration.Config
      • flytekit.configuration.Image
      • flytekit.configuration.ImageConfig
      • flytekit.configuration.SerializationSettings
      • flytekit.configuration.FastSerializationSettings
      • flytekit.configuration.PlatformConfig
      • flytekit.configuration.StatsConfig
      • flytekit.configuration.SecretsConfig
      • flytekit.configuration.S3Config
      • flytekit.configuration.GCSConfig
      • flytekit.configuration.DataConfig
    • ImageSpec
      • flytekit.image_spec.ImageSpec
    • Remote Access
      • flytekit.remote.remote.FlyteRemote
      • flytekit.remote.remote.Options
      • flytekit.remote.entities.FlyteTask
      • flytekit.remote.entities.FlyteWorkflow
      • flytekit.remote.entities.FlyteLaunchPlan
      • flytekit.remote.entities.FlyteNode
      • flytekit.remote.entities.FlyteTaskNode
      • flytekit.remote.entities.FlyteWorkflowNode
      • flytekit.remote.executions.FlyteWorkflowExecution
      • flytekit.remote.executions.FlyteTaskExecution
      • flytekit.remote.executions.FlyteNodeExecution
    • Clients
      • flytekit.clients.friendly.SynchronousFlyteClient
      • flytekit.clients.raw.RawSynchronousFlyteClient
    • Unit Testing
      • flytekit.testing.patch
      • flytekit.testing.task_mock
    • Specifying Accelerators
      • flytekit.extras.accelerators.BaseAccelerator
      • flytekit.extras.accelerators.GPUAccelerator
      • flytekit.extras.accelerators.MultiInstanceGPUAccelerator
      • flytekit.extras.accelerators.A10G
      • flytekit.extras.accelerators.L4
      • flytekit.extras.accelerators.K80
      • flytekit.extras.accelerators.M60
      • flytekit.extras.accelerators.P4
      • flytekit.extras.accelerators.P100
      • flytekit.extras.accelerators.T4
      • flytekit.extras.accelerators.V100
      • flytekit.extras.accelerators.A100
      • flytekit.extras.accelerators.A100_80GB
    • Extending Flytekit
      • flytekit.extend.get_serializable
      • flytekit.extend.context_manager
      • flytekit.extend.IgnoreOutputs
      • flytekit.extend.ExecutionState
      • flytekit.extend.Image
      • flytekit.extend.ImageConfig
      • flytekit.extend.Interface
      • flytekit.extend.Promise
      • flytekit.extend.TaskPlugins
      • flytekit.extend.DictTransformer
      • flytekit.extend.T
      • flytekit.extend.TypeEngine
      • flytekit.extend.TypeTransformer
      • flytekit.extend.PythonCustomizedContainerTask
      • flytekit.extend.ExecutableTemplateShimTask
      • flytekit.extend.ShimTaskExecutor
    • Flyte Deck
      • flytekit.deck.Deck
      • flytekit.deck.TopFrameRenderer
      • flytekit.deck.MarkdownRenderer
      • flytekit.deck.SourceCodeRenderer
    • Plugin API reference
      • AWS Athena
        • flytekitplugins.athena.AthenaConfig
        • flytekitplugins.athena.AthenaTask
      • AWS Batch
        • flytekitplugins.awsbatch.AWSBatchConfig
      • Google Bigquery
        • flytekitplugins.bigquery.BigQueryConfig
        • flytekitplugins.bigquery.BigQueryTask
        • flytekitplugins.bigquery.BigQueryAgent
      • Dask
        • flytekitplugins.dask.Dask
        • flytekitplugins.dask.Scheduler
        • flytekitplugins.dask.WorkerGroup
      • Deck standard
        • flytekitplugins.deck.BoxRenderer
        • flytekitplugins.deck.FrameProfilingRenderer
        • flytekitplugins.deck.MarkdownRenderer
        • flytekitplugins.deck.ImageRenderer
        • flytekitplugins.deck.TableRenderer
      • Dolt standard
        • flytekitplugins.dolt.DoltConfig
        • flytekitplugins.dolt.DoltTable
        • flytekitplugins.dolt.DoltTableNameTransformer
      • Great expectations
        • flytekitplugins.great_expectations.BatchRequestConfig
        • flytekitplugins.great_expectations.GreatExpectationsFlyteConfig
        • flytekitplugins.great_expectations.GreatExpectationsTask
        • flytekitplugins.great_expectations.GreatExpectationsType
      • Hive
        • flytekitplugins.hive.HiveConfig
        • flytekitplugins.hive.HiveSelectTask
        • flytekitplugins.hive.HiveTask
      • Pod
        • flytekitplugins.pod.Pod
      • KF MPI
        • flytekitplugins.kfmpi.MPIJob
      • KF Pytorch
        • flytekitplugins.kfpytorch.PyTorch
        • flytekitplugins.kfpytorch.Elastic
      • KF Tensorflow
        • flytekitplugins.kftensorflow.TfJob
      • Modin
        • flytekitplugins.modin.ModinPandasDataFrameTransformer
      • Pandera
        • flytekitplugins.pandera.PanderaTransformer
      • Papermill
        • flytekitplugins.papermill.NotebookTask
        • flytekitplugins.papermill.record_outputs
      • Snowflake
        • flytekitplugins.snowflake.SnowflakeConfig
        • flytekitplugins.snowflake.SnowflakeTask
        • flytekitplugins.snowflake.SnowflakeAgent
      • Spark
        • flytekitplugins.spark.new_spark_session
        • flytekitplugins.spark.ParquetToSparkDecodingHandler
        • flytekitplugins.spark.Spark
        • flytekitplugins.spark.SparkDataFrameSchemaReader
        • flytekitplugins.spark.SparkDataFrameSchemaWriter
        • flytekitplugins.spark.SparkDataFrameTransformer
      • SQLAlchemy
        • flytekitplugins.sqlalchemy.SQLAlchemyConfig
        • flytekitplugins.sqlalchemy.SQLAlchemyDefaultImages
        • flytekitplugins.sqlalchemy.SQLAlchemyTask
      • ONNX PyTorch
        • flytekitplugins.onnxpytorch.PyTorch2ONNXConfig
        • flytekitplugins.onnxpytorch.PyTorch2ONNX
      • ONNX TensorFlow
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNXConfig
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNX
      • ONNX ScikitLearn
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNXConfig
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNX
      • Ray
        • flytekitplugins.ray.HeadNodeConfig
        • flytekitplugins.ray.RayJobConfig
        • flytekitplugins.ray.WorkerNodeConfig
      • DBT
        • flytekitplugins.dbt.DBTRun
        • flytekitplugins.dbt.DBTTest
        • flytekitplugins.dbt.BaseDBTInput
        • flytekitplugins.dbt.BaseDBTOutput
        • flytekitplugins.dbt.DBTRunInput
        • flytekitplugins.dbt.DBTRunOutput
        • flytekitplugins.dbt.DBTTestInput
        • flytekitplugins.dbt.DBTTestOutput
      • Vaex
        • flytekitplugins.vaex.VaexDataFrameToParquetEncodingHandler
        • flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler
      • MLflow
        • flytekitplugins.mlflow.mlflow_autolog
      • DuckDB
        • flytekitplugins.duckdb.DuckDBQuery
      • SageMaker Inference
        • flytekitplugins.awssagemaker_inference.BotoAgent
        • flytekitplugins.awssagemaker_inference.BotoTask
        • flytekitplugins.awssagemaker_inference.SageMakerModelTask
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointConfigTask
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointAgent
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteEndpointConfigTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteEndpointTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteModelTask
        • flytekitplugins.awssagemaker_inference.SageMakerInvokeEndpointTask
        • flytekitplugins.awssagemaker_inference.create_sagemaker_deployment
        • flytekitplugins.awssagemaker_inference.delete_sagemaker_deployment
      • OpenAI
        • flytekitplugins.openai.BatchEndpointAgent
        • flytekitplugins.openai.BatchEndpointTask
        • flytekitplugins.openai.BatchResult
        • flytekitplugins.openai.DownloadJSONFilesTask
        • flytekitplugins.openai.UploadJSONLFileTask
        • flytekitplugins.openai.OpenAIFileConfig
        • flytekitplugins.openai.create_batch
        • flytekitplugins.openai.ChatGPTAgent
        • flytekitplugins.openai.ChatGPTTask
    • Custom Tasks
      • flytekit.core.base_task.kwtypes
      • flytekit.core.base_task.PythonTask
      • flytekit.core.base_task.Task
      • flytekit.core.base_task.TaskResolverMixin
      • flytekit.core.base_task.IgnoreOutputs
      • flytekit.core.python_function_task.PythonFunctionTask
      • flytekit.core.python_function_task.PythonInstanceTask
      • Shell Task
        • flytekit.extras.tasks.shell.ShellTask
        • flytekit.extras.tasks.shell.OutputLocation
      • SQLite3 Task
        • flytekit.extras.sqlite3.task.SQLite3Task
        • flytekit.extras.sqlite3.task.SQLite3Config
    • Custom Types
      • Flytekit StructuredDataset
        • flytekit.types.structured.StructuredDataset
        • flytekit.types.structured.StructuredDatasetEncoder
        • flytekit.types.structured.StructuredDatasetDecoder
      • Flytekit File Type
        • flytekit.types.file.FlyteFile
        • flytekit.types.file.HDF5EncodedFile
        • flytekit.types.file.HTMLPage
        • flytekit.types.file.JoblibSerializedFile
        • flytekit.types.file.JPEGImageFile
        • flytekit.types.file.PDFFile
        • flytekit.types.file.PNGImageFile
        • flytekit.types.file.PythonPickledFile
        • flytekit.types.file.PythonNotebook
        • flytekit.types.file.SVGImageFile
      • Flytekit Directory Type
        • flytekit.types.directory.FlyteDirectory
        • flytekit.types.directory.TensorboardLogs
        • flytekit.types.directory.TFRecordsDirectory
      • Flytekit Iterator Type
        • flytekit.types.iterator.FlyteIterator
        • flytekit.types.iterator.JSON
      • PyTorch Type
        • flytekit.extras.pytorch.PyTorchCheckpoint
        • flytekit.extras.pytorch.PyTorchCheckpointTransformer
        • flytekit.extras.pytorch.PyTorchModuleTransformer
        • flytekit.extras.pytorch.PyTorchTensorTransformer
      • TensorFlow Type
        • flytekit.extras.tensorflow.TensorFlowRecordFileTransformer
        • flytekit.extras.tensorflow.TensorFlowRecordsDirTransformer
      • Sklearn Type
        • flytekit.extras.sklearn.SklearnEstimatorTransformer
    • Experimental Features
      • flytekit.experimental.map_task
      • flytekit.experimental.eager
      • flytekit.experimental.EagerException
    • Pyflyte CLI
    • Flytekit Contribution Guide
  • flytectl
    • Overview
    • CLI Entrypoint
    • Verbs
      • flytectl create
      • flytectl completion
      • flytectl get
      • flytectl update
      • flytectl delete
      • flytectl register
      • flytectl config
      • flytectl compile
      • flytectl sandbox
      • flytectl demo
      • flytectl version
      • flytectl upgrade
    • Nouns
      • Project
        • flytectl create project
        • flytectl get project
        • flytectl update project
      • Execution
        • flytectl create execution
        • flytectl get execution
        • flytectl update execution
        • flytectl delete execution
      • Workflow
        • flytectl get workflow
        • flytectl update workflow-meta
      • Task
        • flytectl get task
        • flytectl update task-meta
      • Task resource attribute
        • flytectl get task-resource-attribute
        • flytectl update task-resource-attribute
        • flytectl delete task-resource-attribute
      • Cluster resource attribute
        • flytectl get cluster-resource-attribute
        • flytectl delete cluster-resource-attribute
        • flytectl update cluster-resource-attribute
      • Execution cluster label
        • flytectl get execution-cluster-label
        • flytectl update execution-cluster-label
        • flytectl delete execution-cluster-label
      • Execution queue attribute
        • flytectl get execution-queue-attribute
        • flytectl delete execution-queue-attribute
        • flytectl update execution-queue-attribute
      • Plugin override
        • flytectl get plugin-override
        • flytectl update plugin-override
        • flytectl delete plugin-override
      • Launchplan
        • flytectl get launchplan
        • flytectl update launchplan
        • flytectl update launchplan-meta
      • Workflow execution config
        • flytectl get workflow-execution-config
        • flytectl update workflow-execution-config
        • flytectl delete workflow-execution-config
      • Examples
        • flytectl register examples
      • Files
        • flytectl register files
      • Config
        • flytectl config validate
        • flytectl config init
        • flytectl config docs
        • flytectl config discover
      • Sandbox
        • flytectl sandbox start
        • flytectl sandbox status
        • flytectl sandbox teardown
        • flytectl sandbox exec
      • Demo
        • flytectl demo start
        • flytectl demo status
        • flytectl demo teardown
        • flytectl demo exec
        • flytectl demo reload
    • Contributing Guide
  • flyteidl
    • Flyte Core Language Specification
    • FlyteAdmin Service
    • FlyteAdmin REST and gRPC Interface
    • Data Catalog Service
    • Internal and External Eventing Interface
    • Flyte Task Plugin Specification
    • Contributing Guide

Ecosystem

  • flytekit-java
  • unionml
  • pterodactyl
  • latch sdk

Community

  • Community Resources
  • Contributing to Flyte
  • Contributing Examples
  • Roadmap
  • Frequently Asked Questions
  • Troubleshooting Guide

Glossary

  • Main Concepts
    • Tasks
    • Workflows
    • Nodes
    • Launch plans
    • Schedules
    • Registration
    • Executions
    • Understanding the State Transition in a Workflow
    • Timeline of a workflow execution
    • Understand How Flyte Handles Data
    • How to Use Flyte UI
    • What is Data Catalog?
    • Versions
    • Understand the Lifecycle of a Flyte Workflow
  • Control Plane
    • Projects
    • Domains
    • FlyteAdmin
    • FlyteConsole
    • Dynamic Job Spec
  • Component Architecture
    • FlytePropeller Architecture
    • Flyte Native Scheduler Architecture
Flyte
Back to top
Edit this page

Configure AWS PluginsΒΆ

Tags: AWS, Integration, MachineLearning, Data, Advanced

Discover the process of setting up AWS plugins for Flyte.

AWS Batch

Guide to setting up the AWS Batch plugin.

AWS Athena

Guide to setting up the AWS Athena plugin.

AWS Sagemaker

Guide to setting up the AWS Sagemaker plugin.

Next
AWS Batch
Previous
Configure Kubernetes Plugins
Copyright © 2022, Flyte Authors
flytekit: dev, 3cd5876
flytesnacks: dev, 5aee3d0
flytectl: ,