Contents Menu Expand Light mode Dark mode Auto light/dark mode Auto light/dark, in light mode Auto light/dark, in dark mode
Flyte
Logo
Flyte
  • Introduction
  • Quickstart guide
  • Getting started with workflow development
    • Installing development tools
    • Creating a Flyte project
    • Flyte project components
    • Running a workflow locally
  • Flyte fundamentals
    • Tasks, workflows and launch plans
    • Registering workflows
    • Running and scheduling workflows
    • Visualizing task input and output
    • Optimizing tasks
    • Extending Flyte
  • Flyte agents
    • Testing agents in a local Python environment
    • Enabling agents in your Flyte deployment
    • Developing agents
    • Testing agents in a local development cluster
    • Deploying agents to the Flyte sandbox
    • Implementing the agent metadata service
    • How Secret Works in Agent
  • Core use cases
    • Data engineering
    • Machine learning
    • Analytics

Examples

  • User Guide
    • Environment setup
    • Basics
      • Hello, World!
      • Tasks
      • Workflows
      • Launch plans
      • Imperative workflows
      • Documenting workflows
      • Shell tasks
      • Named outputs
    • Data Types and IO
      • FlyteFile
      • FlyteDirectory
      • StructuredDataset
      • Dataclass
      • Accessing attributes
      • PyTorch type
      • Enum type
      • Pickle type
    • Advanced composition
      • Conditionals
      • Chaining Flyte entities
      • Subworkflows
      • Dynamic workflows
      • Map tasks
      • Nested parallelization
      • Eager workflows
      • Decorating tasks
      • Decorating workflows
      • Intratask checkpoints
      • Waiting for external inputs
    • Customizing dependencies
      • ImageSpec
      • Raw containers
      • Multiple images in a workflow
    • Development lifecycle
      • Private images
      • Caching
      • Cache serializing
      • Decks
      • Failure node
      • Creating a new project
      • Running tasks
      • Running workflows
      • Running launch plans
      • Inspecting executions
      • Debugging executions
      • Migrating from Airflow to Flyte
      • For current Flyte users
    • Testing
      • Mocking tasks
    • Productionize
      • Customizing task resources
      • Reference tasks
      • Reference launch plans
      • Notifications
      • Schedules
      • Configuring logging links in the UI
      • Configuring access to GPUs
      • Spot instances
      • Secrets
      • Workflow labels and annotations
    • Extending Flyte
      • Custom types
      • Prebuilt container task plugins
      • User container task plugins
      • Backend plugins
      • Container interface
  • Tutorials
    • Diabetes Classification
      • Train and Validate a Diabetes Classification XGBoost Model
    • House Price Regression
      • Predicting House Price in a Region Using XGBoost
      • Predicting House Price in Multiple Regions Using XGBoost and Dynamic Workflows
    • MNIST Classification With PyTorch and W&B
      • Single Node, Single GPU Training
      • Single Node, Multi GPU Training
    • NLP Processing
      • Word Embeddings and Topic Modelling with Gensim
    • Forecasting Rossman Store Sales with Horovod and Spark
      • Data-Parallel Distributed Training Using Horovod on Spark
    • EDA, Feature Engineering, and Modeling With Papermill
      • Flyte Pipeline in One Jupyter Notebook
      • EDA and Feature Engineering in Jupyter Notebook and Modeling in a Flyte Task
      • EDA and Feature Engineering in One Jupyter Notebook and Modeling in the Other
      • Supermarket Regression Notebook
      • Supermarket Regression 1 Notebook
      • Supermarket Regression 2 Notebook
    • Feast Integration
      • Feature Engineering Tasks
      • Flyte Pipeline with Feast
      • How to Trigger the Feast Workflow using FlyteRemote
    • Nucleotide Sequence Querying with BLASTX
      • BLASTX Example
    • Weather Forecasting
  • Integrations
    • SQL
      • Sqlite3
      • SQLAlchemy
    • Great Expectations
      • Task Example
      • Type Example
    • Papermill
      • Jupyter Notebook Tasks
      • Simple Papermill Notebook
    • Pandera
      • Basic Schema Example
      • Validating and Testing Machine Learning Pipelines
    • Modin
      • KNN Classifier
    • Dolt
      • Quickstart
      • Dolt Branches
    • DBT
      • dbt example
    • WhyLogs
      • whylogs Example
    • MLFlow
      • MLflow Example
    • ONNX
      • PyTorch Example
      • ScikitLearn Example
      • TensorFlow Example
    • DuckDB
      • DuckDB Example
    • Weights and Biases
      • Weights and Biases Example
    • NIM
      • Serve Generative AI Models with NIM
    • Kubernetes Pods
      • Pod Example
    • Dask
      • Running a Dask Task
    • Spark
      • Running a Spark Task
      • Converting a Spark DataFrame to a Pandas DataFrame
    • PyTorch Distributed
      • Run PyTorch Distributed
      • Use PyTorch Lightning to Train an MNIST Autoencoder
    • TensorFlow Distributed
      • Run Distributed TensorFlow Training
    • MPI
      • Running Distributed Training Using Horovod and MPI
    • Ray
      • Running Ray Tasks
    • Airflow agent
      • Airflow agent example usage
    • BigQuery agent
      • BigQuery agent example usage
    • ChatGPT agent
      • ChatGPT agent example usage
    • Databricks agent example
      • Running Spark on Databricks
    • Memory Machine Cloud agent
      • Memory Machine Cloud agent example usage
    • OpenAI Batch Agent
      • Batching Requests for Asynchronous Processing
    • AWS SageMaker Inference Agent
      • Deploy and serve an XGBoost model on AWS SageMaker using FastAPI
    • Sensor
      • File Sensor
    • Snowflake agent
      • Querying data in Snowflake
    • AWS Athena
      • Athena Query
    • AWS Batch
      • AWS Batch
    • FlyteInteractive
      • FlyteInteractive Vscode Decorator
      • FlyteInteractive Jupyter Decorator
    • Hive
      • Hive Tasks
    • Airflow Provider
      • FlyteOperator Example
  • Deprecated integrations
    • BigQuery plugin
      • BigQuery example query
    • Databricks plugin
      • Running Spark on Databricks
    • Snowflake plugin
      • Querying data in Snowflake

Cluster Deployment

  • Getting Started
  • Deployment Paths
    • Sandbox Deployment
    • Single Cluster Simple Cloud Deployment
    • Single Cluster Production-grade Cloud Deployment
    • Multiple Kubernetes Cluster Deployment
  • Plugin Setup
    • Configure Kubernetes Plugins
    • Configure AWS Plugins
      • AWS Batch
      • Athena Plugin
      • Sagemaker Plugin Setup
    • Configure GCP Plugins
      • Google BigQuery Plugin
    • Configure Web APIs
      • Snowflake Plugin
      • Databricks Plugin
  • Agent Setup
    • Airflow agent
    • ChatGPT agent
    • Databricks agent
    • Google BigQuery agent
    • MMCloud Agent
    • SageMaker Inference Agent
    • Sensor agent
    • Snowflake agent
    • OpenAI Batch Agent
  • Configuration
    • Authenticating in Flyte
    • Migrating Your Authentication Config
    • Understanding Authentication
    • Configuring task pods with K8s PodTemplates
    • Customizing project, domain, and workflow resources with flytectl
    • Monitoring
    • Notifications
    • Optimizing Performance
    • Cloud Events
    • Flyte ResourceManager
  • Configuration Reference
    • Flyte Datacatalog Configuration
    • Flyte Admin Configuration
    • Flyte Propeller Configuration
    • Flyte Scheduler Configuration
  • Security Overview
  • Flyte API Playground: Swagger

API Reference

  • flytekit
    • Overview
      • Model Files
      • Authoring Structure
      • Control Plane: FlyteRemote
      • Execution Time Support
      • Command Line Interfaces and Clients
    • Core Flytekit
      • flytekit.task
      • flytekit.workflow
      • flytekit.kwtypes
      • flytekit.current_context
      • flytekit.ExecutionParameters
      • flytekit.FlyteContext
      • flytekit.map_task
      • flytekit.core.workflow.ImperativeWorkflow
      • flytekit.core.node_creation.create_node
      • flytekit.core.promise.NodeOutput
      • flytekit.FlyteContextManager
      • flytekit.conditional
      • flytekit.TaskMetadata
      • flytekit.Resources
      • flytekit.WorkflowFailurePolicy
      • flytekit.PodTemplate
      • flytekit.dynamic
      • flytekit.approve
      • flytekit.sleep
      • flytekit.wait_for_input
      • flytekit.CronSchedule
      • flytekit.FixedRate
      • flytekit.Email
      • flytekit.PagerDuty
      • flytekit.Slack
      • flytekit.get_reference_entity
      • flytekit.LaunchPlanReference
      • flytekit.TaskReference
      • flytekit.WorkflowReference
      • flytekit.reference_task
      • flytekit.reference_workflow
      • flytekit.reference_launch_plan
      • flytekit.SQLTask
      • flytekit.ContainerTask
      • flytekit.PythonFunctionTask
      • flytekit.PythonInstanceTask
      • flytekit.LaunchPlan
      • flytekit.Secret
      • flytekit.SecurityContext
      • flytekit.AuthRole
      • flytekit.Labels
      • flytekit.Annotations
      • flytekit.WorkflowExecutionPhase
      • flytekit.Blob
      • flytekit.BlobMetadata
      • flytekit.Literal
      • flytekit.Scalar
      • flytekit.LiteralType
      • flytekit.BlobType
      • flytekit.HashMethod
      • flytekit.Artifact
      • flytekit.Description
      • flytekit.Documentation
      • flytekit.SourceCode
    • Configuration
      • flytekit.configuration.Config
      • flytekit.configuration.Image
      • flytekit.configuration.ImageConfig
      • flytekit.configuration.SerializationSettings
      • flytekit.configuration.FastSerializationSettings
      • flytekit.configuration.PlatformConfig
      • flytekit.configuration.StatsConfig
      • flytekit.configuration.SecretsConfig
      • flytekit.configuration.S3Config
      • flytekit.configuration.GCSConfig
      • flytekit.configuration.DataConfig
    • ImageSpec
      • flytekit.image_spec.ImageSpec
    • Remote Access
      • flytekit.remote.remote.FlyteRemote
      • flytekit.remote.remote.Options
      • flytekit.remote.entities.FlyteTask
      • flytekit.remote.entities.FlyteWorkflow
      • flytekit.remote.entities.FlyteLaunchPlan
      • flytekit.remote.entities.FlyteNode
      • flytekit.remote.entities.FlyteTaskNode
      • flytekit.remote.entities.FlyteWorkflowNode
      • flytekit.remote.executions.FlyteWorkflowExecution
      • flytekit.remote.executions.FlyteTaskExecution
      • flytekit.remote.executions.FlyteNodeExecution
    • Clients
      • flytekit.clients.friendly.SynchronousFlyteClient
      • flytekit.clients.raw.RawSynchronousFlyteClient
    • Unit Testing
      • flytekit.testing.patch
      • flytekit.testing.task_mock
    • Specifying Accelerators
      • flytekit.extras.accelerators.BaseAccelerator
      • flytekit.extras.accelerators.GPUAccelerator
      • flytekit.extras.accelerators.MultiInstanceGPUAccelerator
      • flytekit.extras.accelerators.A10G
      • flytekit.extras.accelerators.L4
      • flytekit.extras.accelerators.K80
      • flytekit.extras.accelerators.M60
      • flytekit.extras.accelerators.P4
      • flytekit.extras.accelerators.P100
      • flytekit.extras.accelerators.T4
      • flytekit.extras.accelerators.V100
      • flytekit.extras.accelerators.A100
      • flytekit.extras.accelerators.A100_80GB
    • Extending Flytekit
      • flytekit.extend.get_serializable
      • flytekit.extend.context_manager
      • flytekit.extend.IgnoreOutputs
      • flytekit.extend.ExecutionState
      • flytekit.extend.Image
      • flytekit.extend.ImageConfig
      • flytekit.extend.Interface
      • flytekit.extend.Promise
      • flytekit.extend.TaskPlugins
      • flytekit.extend.DictTransformer
      • flytekit.extend.T
      • flytekit.extend.TypeEngine
      • flytekit.extend.TypeTransformer
      • flytekit.extend.PythonCustomizedContainerTask
      • flytekit.extend.ExecutableTemplateShimTask
      • flytekit.extend.ShimTaskExecutor
    • Flyte Deck
      • flytekit.deck.Deck
      • flytekit.deck.TopFrameRenderer
      • flytekit.deck.MarkdownRenderer
      • flytekit.deck.SourceCodeRenderer
    • Plugin API reference
      • AWS Athena
        • flytekitplugins.athena.AthenaConfig
        • flytekitplugins.athena.AthenaTask
      • AWS Batch
        • flytekitplugins.awsbatch.AWSBatchConfig
      • Google Bigquery
        • flytekitplugins.bigquery.BigQueryConfig
        • flytekitplugins.bigquery.BigQueryTask
        • flytekitplugins.bigquery.BigQueryAgent
      • Dask
        • flytekitplugins.dask.Dask
        • flytekitplugins.dask.Scheduler
        • flytekitplugins.dask.WorkerGroup
      • Deck standard
        • flytekitplugins.deck.BoxRenderer
        • flytekitplugins.deck.FrameProfilingRenderer
        • flytekitplugins.deck.MarkdownRenderer
        • flytekitplugins.deck.ImageRenderer
        • flytekitplugins.deck.TableRenderer
      • Dolt standard
        • flytekitplugins.dolt.DoltConfig
        • flytekitplugins.dolt.DoltTable
        • flytekitplugins.dolt.DoltTableNameTransformer
      • Great expectations
        • flytekitplugins.great_expectations.BatchRequestConfig
        • flytekitplugins.great_expectations.GreatExpectationsFlyteConfig
        • flytekitplugins.great_expectations.GreatExpectationsTask
        • flytekitplugins.great_expectations.GreatExpectationsType
      • Hive
        • flytekitplugins.hive.HiveConfig
        • flytekitplugins.hive.HiveSelectTask
        • flytekitplugins.hive.HiveTask
      • Pod
      • KF MPI
        • flytekitplugins.kfmpi.MPIJob
      • KF Pytorch
        • flytekitplugins.kfpytorch.PyTorch
        • flytekitplugins.kfpytorch.Elastic
      • KF Tensorflow
        • flytekitplugins.kftensorflow.TfJob
      • Modin
        • flytekitplugins.modin.ModinPandasDataFrameTransformer
      • Pandera
        • flytekitplugins.pandera.PanderaTransformer
      • Papermill
        • flytekitplugins.papermill.NotebookTask
        • flytekitplugins.papermill.record_outputs
      • Snowflake
        • flytekitplugins.snowflake.SnowflakeConfig
        • flytekitplugins.snowflake.SnowflakeTask
        • flytekitplugins.snowflake.SnowflakeAgent
      • Spark
        • flytekitplugins.spark.new_spark_session
        • flytekitplugins.spark.ParquetToSparkDecodingHandler
        • flytekitplugins.spark.Spark
        • flytekitplugins.spark.SparkDataFrameSchemaReader
        • flytekitplugins.spark.SparkDataFrameSchemaWriter
        • flytekitplugins.spark.SparkDataFrameTransformer
      • SQLAlchemy
        • flytekitplugins.sqlalchemy.SQLAlchemyConfig
        • flytekitplugins.sqlalchemy.SQLAlchemyDefaultImages
        • flytekitplugins.sqlalchemy.SQLAlchemyTask
      • ONNX PyTorch
        • flytekitplugins.onnxpytorch.PyTorch2ONNXConfig
        • flytekitplugins.onnxpytorch.PyTorch2ONNX
      • ONNX TensorFlow
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNXConfig
        • flytekitplugins.onnxtensorflow.TensorFlow2ONNX
      • ONNX ScikitLearn
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNXConfig
        • flytekitplugins.onnxscikitlearn.ScikitLearn2ONNX
      • Ray
        • flytekitplugins.ray.HeadNodeConfig
        • flytekitplugins.ray.RayJobConfig
        • flytekitplugins.ray.WorkerNodeConfig
      • DBT
        • flytekitplugins.dbt.DBTRun
        • flytekitplugins.dbt.DBTTest
        • flytekitplugins.dbt.BaseDBTInput
        • flytekitplugins.dbt.BaseDBTOutput
        • flytekitplugins.dbt.DBTRunInput
        • flytekitplugins.dbt.DBTRunOutput
        • flytekitplugins.dbt.DBTTestInput
        • flytekitplugins.dbt.DBTTestOutput
      • Vaex
        • flytekitplugins.vaex.VaexDataFrameToParquetEncodingHandler
        • flytekitplugins.vaex.ParquetToVaexDataFrameDecodingHandler
      • MLflow
        • flytekitplugins.mlflow.mlflow_autolog
      • DuckDB
        • flytekitplugins.duckdb.DuckDBQuery
      • SageMaker Inference
        • flytekitplugins.awssagemaker_inference.BotoAgent
        • flytekitplugins.awssagemaker_inference.BotoTask
        • flytekitplugins.awssagemaker_inference.SageMakerModelTask
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointConfigTask
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointAgent
        • flytekitplugins.awssagemaker_inference.SageMakerEndpointTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteEndpointConfigTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteEndpointTask
        • flytekitplugins.awssagemaker_inference.SageMakerDeleteModelTask
        • flytekitplugins.awssagemaker_inference.SageMakerInvokeEndpointTask
        • flytekitplugins.awssagemaker_inference.create_sagemaker_deployment
        • flytekitplugins.awssagemaker_inference.delete_sagemaker_deployment
      • OpenAI
        • flytekitplugins.openai.BatchEndpointAgent
        • flytekitplugins.openai.BatchEndpointTask
        • flytekitplugins.openai.BatchResult
        • flytekitplugins.openai.DownloadJSONFilesTask
        • flytekitplugins.openai.UploadJSONLFileTask
        • flytekitplugins.openai.OpenAIFileConfig
        • flytekitplugins.openai.create_batch
        • flytekitplugins.openai.ChatGPTAgent
        • flytekitplugins.openai.ChatGPTTask
      • Inference
        • flytekitplugins.inference.NIM
        • flytekitplugins.inference.NIMSecrets
    • Custom Tasks
      • flytekit.core.base_task.kwtypes
      • flytekit.core.base_task.PythonTask
      • flytekit.core.base_task.Task
      • flytekit.core.base_task.TaskResolverMixin
      • flytekit.core.base_task.IgnoreOutputs
      • flytekit.core.python_function_task.PythonFunctionTask
      • flytekit.core.python_function_task.PythonInstanceTask
      • Shell Task
        • flytekit.extras.tasks.shell.ShellTask
        • flytekit.extras.tasks.shell.OutputLocation
      • SQLite3 Task
        • flytekit.extras.sqlite3.task.SQLite3Task
        • flytekit.extras.sqlite3.task.SQLite3Config
    • Custom Types
      • Flytekit StructuredDataset
        • flytekit.types.structured.StructuredDataset
        • flytekit.types.structured.StructuredDatasetEncoder
        • flytekit.types.structured.StructuredDatasetDecoder
      • Flytekit File Type
        • flytekit.types.file.FlyteFile
        • flytekit.types.file.HDF5EncodedFile
        • flytekit.types.file.HTMLPage
        • flytekit.types.file.JoblibSerializedFile
        • flytekit.types.file.JPEGImageFile
        • flytekit.types.file.PDFFile
        • flytekit.types.file.PNGImageFile
        • flytekit.types.file.PythonPickledFile
        • flytekit.types.file.PythonNotebook
        • flytekit.types.file.SVGImageFile
      • Flytekit Directory Type
        • flytekit.types.directory.FlyteDirectory
        • flytekit.types.directory.TensorboardLogs
        • flytekit.types.directory.TFRecordsDirectory
      • Flytekit Iterator Type
        • flytekit.types.iterator.FlyteIterator
        • flytekit.types.iterator.JSON
      • PyTorch Type
        • flytekit.extras.pytorch.PyTorchCheckpoint
        • flytekit.extras.pytorch.PyTorchCheckpointTransformer
        • flytekit.extras.pytorch.PyTorchModuleTransformer
        • flytekit.extras.pytorch.PyTorchTensorTransformer
      • TensorFlow Type
        • flytekit.extras.tensorflow.TensorFlowRecordFileTransformer
        • flytekit.extras.tensorflow.TensorFlowRecordsDirTransformer
      • Sklearn Type
        • flytekit.extras.sklearn.SklearnEstimatorTransformer
    • Experimental Features
      • flytekit.experimental.map_task
      • flytekit.experimental.eager
      • flytekit.experimental.EagerException
    • Pyflyte CLI
    • Flytekit Contribution Guide
  • flytectl
    • Overview
    • CLI Entrypoint
    • Verbs
      • flytectl create
      • flytectl completion
      • flytectl get
      • flytectl update
      • flytectl delete
      • flytectl register
      • flytectl config
      • flytectl compile
      • flytectl sandbox
      • flytectl demo
      • flytectl version
      • flytectl upgrade
    • Nouns
      • Project
        • flytectl create project
        • flytectl get project
        • flytectl update project
      • Execution
        • flytectl create execution
        • flytectl get execution
        • flytectl update execution
        • flytectl delete execution
      • Workflow
        • flytectl get workflow
        • flytectl update workflow-meta
      • Task
        • flytectl get task
        • flytectl update task-meta
      • Task resource attribute
        • flytectl get task-resource-attribute
        • flytectl update task-resource-attribute
        • flytectl delete task-resource-attribute
      • Cluster resource attribute
        • flytectl get cluster-resource-attribute
        • flytectl delete cluster-resource-attribute
        • flytectl update cluster-resource-attribute
      • Execution cluster label
        • flytectl get execution-cluster-label
        • flytectl update execution-cluster-label
        • flytectl delete execution-cluster-label
      • Execution queue attribute
        • flytectl get execution-queue-attribute
        • flytectl delete execution-queue-attribute
        • flytectl update execution-queue-attribute
      • Plugin override
        • flytectl get plugin-override
        • flytectl update plugin-override
        • flytectl delete plugin-override
      • Launchplan
        • flytectl get launchplan
        • flytectl update launchplan
        • flytectl update launchplan-meta
      • Workflow execution config
        • flytectl get workflow-execution-config
        • flytectl update workflow-execution-config
        • flytectl delete workflow-execution-config
      • Examples
        • flytectl register examples
      • Files
        • flytectl register files
      • Config
        • flytectl config validate
        • flytectl config init
        • flytectl config docs
        • flytectl config discover
      • Sandbox
        • flytectl sandbox start
        • flytectl sandbox status
        • flytectl sandbox teardown
        • flytectl sandbox exec
      • Demo
        • flytectl demo start
        • flytectl demo status
        • flytectl demo teardown
        • flytectl demo exec
        • flytectl demo reload
    • Contributing Guide
  • flyteidl
    • Flyte Core Language Specification
    • FlyteAdmin Service
    • FlyteAdmin REST and gRPC Interface
    • Data Catalog Service
    • Internal and External Eventing Interface
    • Flyte Task Plugin Specification
    • Contributing Guide

Ecosystem

  • flytekit-java
  • unionml
  • pterodactyl
  • latch sdk

Community

  • Community Resources
  • Contributing to Flyte
  • Contributing Examples
  • Roadmap
  • Frequently Asked Questions
  • Troubleshooting Guide

Glossary

  • Main Concepts
    • Tasks
    • Workflows
    • Nodes
    • Launch plans
    • Schedules
    • Registration
    • Executions
    • Understanding the State Transition in a Workflow
    • Timeline of a workflow execution
    • Understand How Flyte Handles Data
    • How to Use Flyte UI
    • What is Data Catalog?
    • Versions
    • Understand the Lifecycle of a Flyte Workflow
  • Control Plane
    • Projects
    • Domains
    • FlyteAdmin
    • FlyteConsole
    • Dynamic Job Spec
  • Component Architecture
    • FlytePropeller Architecture
    • Flyte Native Scheduler Architecture
Flyte
Back to top
Edit this page

Model Inference referenceΒΆ

Tags: Integration, Serving, Inference

NIM

NIMSecrets

param ngc_image_secret:

The name of the Kubernetes secret containing the NGC image pull credentials.
Next
flytekitplugins.inference.NIM
Previous
flytekitplugins.openai.ChatGPTTask
Copyright © 2022, Flyte Authors
flytekit: dev, 392f1d9
flytesnacks: dev, a8fbfea
flytectl: ,