flytekitplugins.inference.NIM

class flytekitplugins.inference.NIM(secrets, image='nvcr.io/nim/meta/llama3-8b-instruct:1.0.0', health_endpoint='v1/health/ready', port=8000, cpu=1, gpu=1, mem='20Gi', shm_size='16Gi', env=None, hf_repo_ids=None, lora_adapter_mem=None)

Methods

Parameters:

• secrets (NIMSecrets)

• image (str)

• health_endpoint (str)

• port (int)

• cpu (int)

• gpu (int)

• mem (str)

• shm_size (str)

• env (dict[str, str] | None)

• hf_repo_ids (list[str] | None)

• lora_adapter_mem (str | None)

setup_nim_pod_template()

Attributes

base_url

pod_template