Volcano Job API 设计与实践多任务规格、状态机与生命周期策略全解析【免费下载链接】volcanoA Cloud Native Batch System (Project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/vol/volcano本文基于 Volcano 官方设计文档 Job API结合当前仓库的类型定义job.go、webhook 校验实现admit_job.go与 Job 控制器状态机factory.go源码系统讲解 VolcanoJob这一批处理核心对象的 API 设计多 Pod 模板、任务输入/输出、阶段状态机、错误处理策略以及与准入校验、Gang 调度、优先级、插件等特性的交互方式。读完本文你将能够独立编写、校验并深入理解一个 Volcano Job 从提交到运行完成的完整生命周期。设计动机与范围Job是 Volcano 中高吞吐量high performance workload类负载的基础对象。传统 Kubernetes 中一个 TensorFlow 训练任务ps/worker 两类角色或 Spark 作业driver/executor 两类角色需要借助多个独立 Workload 拼装无法作为一个整体被调度、回收或重启。Volcano 的Job对象将一组逻辑上关联、生命周期上耦合的 Pod抽象为单一 CRD使其能够整体进入队列、整体 gang 调度、整体故障恢复。该设计文档明确了能力边界范围内In Scope定义 Job 的 API定义 Job 的行为明确 Job 与其他特性的交互范围外Out of Scope卷Volume的完整管理——卷管理不在 Job 管理特性范围内Job 只声明式地引用数据任务间的网络寻址addressing——由其他项目如 svc 插件对应的服务发现机制单独描述。Job 对象总体结构Job的定义遵循 Kubernetes 惯例由Spec期望行为与Status当前状态组成并带TypeMeta/ObjectMeta标准字段。Job的 CRD 注册在batch.volcano.sh/v1alpha1组下从 job.go 的 kubebuilder 注解可以看到其资源路径为jobs、短名为vcjob/vj且支持status子资源// genclient // kubebuilder:object:roottrue // kubebuilder:resource:pathjobs,shortNamevcjob;vj // kubebuilder:subresource:status // Job defines the volcano job. type Job struct { metav1.TypeMeta json:,inline // optional metav1.ObjectMeta json:metadata,omitempty protobuf:bytes,1,opt,namemetadata // Specification of the desired behavior of the volcano job, including the minAvailable // optional Spec JobSpec json:spec,omitempty protobuf:bytes,2,opt,namespec // Current status of the volcano Job // optional Status JobStatus json:status,omitempty protobuf:bytes,3,opt,namestatus }JobList则是标准的列表对象job.go// kubebuilder:object:roottrue // JobList defines the list of jobs. type JobList struct { metav1.TypeMeta json:,inline metav1.ListMeta json:metadata,omitempty protobuf:bytes,1,opt,namemetadata Items []Job json:items protobuf:bytes,2,rep,nameitems }下文按设计文档的脉络逐一展开JobSpec的核心能力。多 Pod 模板spec.tasks高性能计算任务通常包含不同类型的角色例如 TensorFlowps/worker、Sparkdriver/executor。为此Job引入tasks字段设计文档早期写作taskSpecs当前 API 的 json tag 为tasks支持在同一个 Job 内声明多个 Pod 模板// TaskSpec specifies the task specification of Job. type TaskSpec struct { // Name specifies the name of tasks Name string json:name,omitempty protobuf:bytes,1,opt,namename // Replicas specifies the replicas of this TaskSpec in Job Replicas int32 json:replicas,omitempty protobuf:bytes,2,opt,namereplicas // Specifies the pod that will be created for this TaskSpec // when executing a Job Template v1.PodTemplateSpec json:template,omitempty protobuf:bytes,4,opt,nametemplate // Specifies the lifecycle of task // optional Policies []LifecyclePolicy json:policies,omitempty protobuf:bytes,5,opt,namepolicies }JobController依据spec.tasks中每个任务的template与replicas创建 Pod并将所创建 Pod 的OwnerReference指向该Job即 Pod 由 Job 控制、随 Job 回收。一个多任务模板的示例如下一个 2 ps 5 worker 的 TensorFlow 作业apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: tasks: - name: ps replicas: 2 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img从当前源码看TaskSpec 在原始设计之上又扩展了若干字段均可选MinAvailable *int32任务级的最小可用副本数默认等于该任务的ReplicasTopologyPolicyNUMA 拓扑策略枚举值none / best-effort / restricted / single-numa-nodeMaxRetry int32任务级最大重试次数默认 3DependsOn *DependsOn任务间依赖见下文交互章节PartitionPolicy将任务副本划分为若干子组的分区策略需满足Replicas TotalPartitions * PartitionSize。这些扩展对应了 Gang 感知驱逐、任务启动顺序等后续设计文档如 gang-aware-eviction-design.md但name replicas template policies四元组仍是 API 的骨架。Job 输入/输出spec.volumes大多数高性能负载都会处理输入/输出数据。Volcano 用VolumeSpec以声明式方式描述 Job 的数据卷JobController负责把卷挂载进每个 Task 的 Pod// VolumeSpec defines the specification of Volume, e.g. PVC. type VolumeSpec struct { // Path within the container at which the volume should be mounted. Must not contain :. MountPath string json:mountPath protobuf:bytes,1,opt,namemountPath // defined the PVC name // optional VolumeClaimName string json:volumeClaimName,omitempty protobuf:bytes,2,opt,namevolumeClaimName // VolumeClaim defines the PVC used by the VolumeMount. // optional VolumeClaim *v1.PersistentVolumeClaimSpec json:volumeClaim,omitempty protobuf:bytes,3,opt,namevolumeClaim }在JobSpec中通过Volumes []VolumeSpec引用。语义规则源自设计文档spec.volumes为nil时表示用户自行管理数据Job 不做任何卷操作若某个VolumeSpec的volumeClaim内联 PVC 规格与volumeClaimName已有 PVC 名称都为空或对应 PVC 不存在则对每个 Task/Pod 使用emptyDir卷兜底。当前 VolumeSpec 实现还增加了 kubebuilder 校验mountPath必填且不允许包含:volumeClaimName长度上限 253。另外从 admit_job.go 的更新校验逻辑可以看出一个实现细节当volumeClaim内联给出时控制器会回填 PVC 名称并清空volumeClaimName后再做一致性比较——即内联 PVC 的名称由 Job 控制器维护而非用户手工同步。状态与阶段Phases 和 ConditionsJob 的当前状态由status.state承载phase给出生命周期的高层摘要reason与message提供最近一次阶段迁移的原因与人类可读详情。当前实现还附带了lastTransitionTimejob.go// JobState contains details for the current state of the job. type JobState struct { // The phase of Job. // optional Phase JobPhase json:phase,omitempty protobuf:bytes,1,opt,namephase // Unique, one-word, CamelCase reason for the phases last transition. // optional Reason string json:reason,omitempty protobuf:bytes,2,opt,namereason // Human-readable message indicating details about last transition. // optional Message string json:message,omitempty protobuf:bytes,3,opt,namemessage // Last time the condition transit from one phase to another. // optional LastTransitionTime metav1.Time json:lastTransitionTime,omitempty protobuf:bytes,4,opt,namelastTransitionTime }设计文档定义的阶段枚举JobPhase及允许的迁移关系如下。表格中空单元格表示不允许迁移到该目标阶段From \ ToPendingAbortedRunningCompletedTerminatedPending***Aborted**Running****Completed*Terminated*Restarting、Aborting、Terminating是临时态用于避免竞态例如TerminateJobAction触发后会相继收到多个PodEvictedEvent如果 Job 没有先进入Terminating这类中间态这些事件会被重复处理。当前源码中的完整阶段集合job.go在设计文档基础上补充了Completing任务满足完成条件、开始清理与Failed重启次数达到maxRetry上限const ( // Pending is the phase that job is pending in the queue, waiting for scheduling decision Pending JobPhase Pending // Aborting is the phase that job is aborted, waiting for releasing pods Aborting JobPhase Aborting // Aborted is the phase that job is aborted by user or error handling Aborted JobPhase Aborted // Running is the phase that minimal available tasks of Job are running Running JobPhase Running // Restarting is the phase that the Job is restarted, waiting for pod releasing and recreating Restarting JobPhase Restarting // Completing is the phase that required tasks of job are completed, job starts to clean up Completing JobPhase Completing // Completed is the phase that all tasks of Job are completed Completed JobPhase Completed // Terminating is the phase that the Job is terminated, waiting for releasing pods Terminating JobPhase Terminating // Terminated is the phase that the job is finished unexpected, e.g. events Terminated JobPhase Terminated // Failed is the phase that the job is restarted failed reached the maximum number of retries. Failed JobPhase Failed )从源码结构看阶段并非自由迁移而是由 Job 控制器中的状态机严格驱动factory.go 的NewState函数按当前phase分派到对应的状态处理器每个状态目录一一对应 pkg/controllers/job/state/ 下的实现文件// NewState gets the state from the volcano job Phase. func NewState(jobInfo *apis.JobInfo) State { job : jobInfo.Job switch job.Status.State.Phase { case vcbatch.Pending: return pendingState{job: jobInfo} case vcbatch.Running: return runningState{job: jobInfo} case vcbatch.Restarting: return restartingState{job: jobInfo} case vcbatch.Terminated, vcbatch.Completed, vcbatch.Failed: return finishedState{job: jobInfo} case vcbatch.Terminating: return terminatingState{job: jobInfo} case vcbatch.Aborting: return abortingState{job: jobInfo} case vcbatch.Aborted: return abortedState{job: jobInfo} case vcbatch.Completing: return completingState{job: jobInfo} } // Its pending by default. return pendingState{job: jobInfo} }State接口统一暴露Execute(act Action)方法各状态内部只接受符合迁移表的ActionAbortJob、RestartJob、TerminateJob、ResumeJob、CompleteJob、SyncJob等从而在代码层面保证了设计文档中阶段迁移矩阵的约束。未识别的阶段默认按Pending处理。JobStatus除state外还提供各阶段 Pod 计数。当前实现job.go的完整字段为字段含义state当前JobStatephase reason message lastTransitionTimeminAvailable该 Job 实际生效的最小可用 Pod 数taskStatusCount按任务名统计的各 Pod 阶段计数map: taskName → phase 计数pending/running/succeeded/failed各阶段 Pod 总数terminating/unknownTerminating / Unknown 阶段 Pod 数version作业当前版本重启递增retryCountJob 累计重试次数配合maxRetryrunningDuration从 Running 到完成的时长controlledResourcesJob 控制的其他资源如 Service、ConfigMap清单conditions导致当前状态的JobCondition列表含status与lastTransitionTime错误处理LifecyclePolicyJob 创建后会发生多种事件Pod 成功、Pod 失败、Pod 被驱逐等其中部分事件对 Job 是致命的如 MPI Job 中任一 Pod 失败。为此引入LifecyclePolicy以事件 → 动作的形式让用户按框架语义配置容错行为。事件Event枚举当前实现在 events.goconst ( // AllEvents means all event AllEvents Event * // PodFailedEvent is triggered if Pod was failed PodFailedEvent Event PodFailed // PodEvictedEvent is triggered if Pod was deleted PodEvictedEvent Event PodEvicted // JobUnknownEvent is triggered when part of pods cant be scheduled // while some are already running in gang-scheduling case JobUnknownEvent Event Unknown // OutOfSyncEvent is triggered if Pod/Job were updated OutOfSyncEvent Event OutOfSync // CommandIssuedEvent is triggered if a command is raised by user CommandIssuedEvent Event CommandIssued // TaskCompletedEvent is triggered if the Replicas amount of pods in one task are succeed TaskCompletedEvent Event TaskCompleted )动作Action枚举当前实现在 actions.goconst ( // AbortJobAction: all Pod of Job will be evicted, and no Pod will be recreated // (the job can be resumed later via ResumeJob) AbortJobAction Action AbortJob // RestartJobAction: the whole job will be restarted RestartJobAction Action RestartJob // TerminateJobAction: the job is terminated and can not be resumed TerminateJobAction Action TerminateJob // CompleteJobAction: unfinished pods will be killed, job completed CompleteJobAction Action CompleteJob // ResumeJobAction: resume an aborted job ResumeJobAction Action ResumeJob // SyncJobAction: sync Job/Pod status SyncJobAction Action SyncJob )策略结构体LifecyclePolicyjob.go// LifecyclePolicy specifies the lifecycle and error handling of task and job. type LifecyclePolicy struct { // The action that will be taken to the PodGroup according to Event. Action v1alpha1.Action json:action,omitempty protobuf:bytes,1,opt,nameaction // The Event recorded by scheduler; the controller takes actions according to this Event. Event v1alpha1.Event json:event,omitempty protobuf:bytes,2,opt,nameevent // The Events recorded by scheduler (multiple events per policy). // optional Events []v1alpha1.Event json:events,omitempty protobuf:bytes,3,opt,nameevents // The exit code of the pod container, controller will take action according to this code. // Note: only one of Event or ExitCode can be specified. // optional ExitCode *int32 json:exitCode,omitempty protobuf:bytes,4,opt,nameexitCode // Timeout is the grace period for controller to take actions. // Default to nil (take action immediately). // optional Timeout *metav1.Duration json:timeout,omitempty protobuf:bytes,5,opt,nametimeout }JobSpec与TaskSpec都包含policies字段二者是默认值 覆盖关系spec.policies是 Job 级默认策略spec.tasks[i].policies会覆盖对应任务的默认策略。示例一机器学习训练任务任一任务失败/被驱逐即重启整个 Job。由于所有任务都不单独设置策略全部任务都继承 Job 级event: * → RestartJobapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: # If any event here, restart the whole job. policies: - event: * action: RestartJob tasks: - name: ps replicas: 1 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img示例二Spark 这类大数据框架的差异化容错。driver 失败则重启整个 Jobexecutor 失败则仅靠 Pod 自身的restartPolicy: OnFailure原地重启不牵连整个作业apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: spark-job spec: tasks: - name: driver replicas: 1 policies: - event: * action: RestartJob template: spec: containers: - name: driver image: driver-img - name: executor replicas: 5 template: spec: containers: - name: executor image: executor-img restartPolicy: OnFailure值得注意的演进点当前实现除了Event还支持按容器退出码ExitCode触发策略以及一条策略关联多个事件Events。此外spec.maxRetry默认 3job.go限制了重启这类动作的总次数超过后 Job 进入Failed阶段——这是设计文档发布后为RestartJob引入的熔断机制。与其他特性的交互Admission 校验设计文档要求准入校验必须保证以下预期行为spec.minAvailable sum(spec.tasks.replicas)spec.tasks数组中无重复的 task 名LifecyclePolicy数组Job 级与 Task 级中无重复的事件处理器。当前实现在 admit_job.go 的validateJobCreate中落地并通过 ValidatingWebhook/jobs/validatewebhook 名validatejob.volcano.sh拦截 Create/Update 请求。除上述三项外实际校验集合还包括未声明任何 task 时直接拒绝No task specified in job spec启用了 MPI 插件时必须能按插件参数找到 master 任务The specified mpi master task was not found生成的 Pod 名job-taskName-index规则必须通过 Kubernetes 限定名长度校验validateK8sPodNameLength每个 task 的template走 Kubernetes 原生ValidatePodTemplate校验spec.plugins中的插件名必须在插件注册表中存在unable to find job plugin引用的spec.queue必须存在且状态为Open且不能提交到root队列或非叶子队列层级队列场景若任务间声明了DependsOn依赖依赖图必须是有向无环图DAG否则拒绝。更新操作的约束validateJobUpdate更为严格不能增删 task除minAvailable、tasks[*].replicas与PriorityClassName之外的 spec 字段不可变更——从源码结构看这正是当前作业扩缩容能力scale up/down所允许的唯一可变面。CoSchedulingGang 调度Gang 调度是 TF、MPI 等负载的刚需要么全体就绪要么全体不启动。spec.minAvailable指明至少多少个 Pod 一起被调度默认值为 sum(spec.tasks.replicas)准入 webhook 校验spec.minAvailable不得超过总副本数否则拒绝创建若spec.minAvailable 总副本数超出的 Pod 会被随机创建如需按序创建应使用任务优先级机制。示例1 ps 5 workerminAvailable: 6即全量 gangapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: # minAvailable to run job minAvailable: 6 tasks: - name: ps replicas: 1 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img任务内优先级Task Priority within Job除多模板外同一 Job 内不同任务还可以有不同优先级。Volcano 复用 Pod 模板的PriorityClass表达任务优先级。示例Spark 作业 1 driver 5 executordriver 使用更高优先级的master-priminAvailable: 3表示资源不足时调度器保证1 driver 2 executor也能先成 gang 启动apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: spark-job spec: minAvailable: 3 tasks: - name: driver replicas: 1 template: spec: priorityClass: master-pri containers: - name: driver image: driver-img - name: executor replicas: 5 template: spec: containers: - name: executor image: executor-img设计文档特别提示了一个竞态虽然调度器会优先调度高优先级 Pod但不同 kubelet 之间仍存在低优先级 Pod 先启动的可能任务间依赖即当前 API 中的spec.tasks[].dependsOn字段见 DependsOn 定义正是为处理这类竞态而引入的后继能力。作业间资源共享默认spec.minAvailable等于总副本数即全量 gang若显式调小minAvailable超出该值的 Pod 将在作业之间共享资源无需独占提升集群利用率。示例与上一节相同minAvailable: 3时第 4~6 个 executor 属于可共享部分。Job 插件spec.pluginsTensorFlow、MPI、MxNet 等框架作业还需要设置环境变量、任务间通信、免密 SSH 登录等胶水工作。Volcano 提供 Job API 插件让用户聚焦核心业务。设计文档发布时提供了三个内置插件每个插件都有参数未提供时使用默认值env向每个容器注入VK_TASK_INDEX当前实现同时注入VC_TASK_INDEX作为容器身份索引——见 env 插件 及其常量定义 const.gosvc为任务创建 Service 与*.host域名使 Pod 之间可通信ssh配置免密 SSH支持mpirun/mpiexec等命令。插件以map[插件名]参数列表挂在spec.plugins上完整示例MPI 作业master 驱逐后重启整个 JobapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: mpi-job spec: minAvailable: 2 schedulerName: volcano policies: - event: PodEvicted action: RestartJob plugins: ssh: [] env: [] svc: [] tasks: - replicas: 1 name: mpimaster template: spec: containers: image: mpi-image name: mpimaster - replicas: 2 name: mpiworker template: spec: containers: image: mpi-image name: mpiworker插件的可用性与参数语义有专门的用户指南如何配置插件、SSH 插件、SVC 插件、ENV 插件、MPI 插件。附录JobSpec完整字段速查以当前源码 job.go 为准JobSpec的全部字段及说明如下设计文档列标注该字段是否出自原始 Job API 设计字段类型 / 默认值说明设计文档schedulerNamestring最大 63 字符tasks.template.spec.schedulerName的默认值是minAvailableint32默认各 task 副本之和该 Job 至少需要同时调度的 Pod 数Gang 规模是volumes[]VolumeSpecJob 级输入/输出卷声明mountPath PVC 名称或内联 PVC 规格是tasks[]TaskSpecMinItems1任务列表每项含 name/replicas/template/policies 及扩展字段是policies[]LifecyclePolicyJob 级默认生命周期策略可被 TaskSpec.policies 覆盖是pluginsmap[string][]string插件名 → 参数列表env/svc/ssh/mpi 等是queuestring默认default作业提交的调度队列是maxRetryint32默认 3作业重试上限超限进入Failed是runningEstimate*Duration用户预估的运行时长否ttlSecondsAfterFinished*int32Completed/Failed 后自动删除 Job 的宽限期否priorityClassNamestring作业级优先级类否minSuccess*int32最小 1最小成功 Pod 数完成条件min-success 特性否networkTopology*NetworkTopologySpec网络拓扑约束mode: hard/soft可配最高允许跨层否TaskSpec扩展字段设计文档之外的演进字段说明minAvailable任务级最小可用副本数默认等于该任务replicastopologyPolicyNUMA 策略none/best-effort/restricted/single-numa-nodemaxRetry任务级重试上限默认 3dependsOn依赖的其他任务名列表 iterationany/all构成 DAG 启动顺序partitionPolicy分区策略totalPartitions、partitionSize、minPartitions、嵌套networkTopology一个可直接参考的完整 Job 样例见 example/job.yaml。小结Volcano 的JobAPI 围绕一个核心思想展开把一组逻辑耦合的 Pod提升为一等调度对象。其骨架由四部分组成——多模板tasks表达异构角色、volumes声明式接入数据、minAvailable定义 Gang 边界、policiesLifecyclePolicy把 Pod 级事件翻译为作业级动作。围绕骨架准入 webhook 保证提交的合法性控制器内的状态机state 目录严格执行阶段迁移表内置插件消化框架胶水逻辑。原始设计文档中关于任务间依赖处理 kubelet 竞态的展望也已落地为当前的dependsOn字段并在 webhook 中做 DAG 校验。理解这套 API 与源码的对应关系是编写、排障和扩展 Volcano 批处理负载的基础。【免费下载链接】volcanoA Cloud Native Batch System (Project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/vol/volcano创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Volcano Job API 设计与实践：多任务规格、状态机与生命周期策略全解析
Volcano Job API 设计与实践多任务规格、状态机与生命周期策略全解析【免费下载链接】volcanoA Cloud Native Batch System (Project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/vol/volcano本文基于 Volcano 官方设计文档 Job API结合当前仓库的类型定义job.go、webhook 校验实现admit_job.go与 Job 控制器状态机factory.go源码系统讲解 VolcanoJob这一批处理核心对象的 API 设计多 Pod 模板、任务输入/输出、阶段状态机、错误处理策略以及与准入校验、Gang 调度、优先级、插件等特性的交互方式。读完本文你将能够独立编写、校验并深入理解一个 Volcano Job 从提交到运行完成的完整生命周期。设计动机与范围Job是 Volcano 中高吞吐量high performance workload类负载的基础对象。传统 Kubernetes 中一个 TensorFlow 训练任务ps/worker 两类角色或 Spark 作业driver/executor 两类角色需要借助多个独立 Workload 拼装无法作为一个整体被调度、回收或重启。Volcano 的Job对象将一组逻辑上关联、生命周期上耦合的 Pod抽象为单一 CRD使其能够整体进入队列、整体 gang 调度、整体故障恢复。该设计文档明确了能力边界范围内In Scope定义 Job 的 API定义 Job 的行为明确 Job 与其他特性的交互范围外Out of Scope卷Volume的完整管理——卷管理不在 Job 管理特性范围内Job 只声明式地引用数据任务间的网络寻址addressing——由其他项目如 svc 插件对应的服务发现机制单独描述。Job 对象总体结构Job的定义遵循 Kubernetes 惯例由Spec期望行为与Status当前状态组成并带TypeMeta/ObjectMeta标准字段。Job的 CRD 注册在batch.volcano.sh/v1alpha1组下从 job.go 的 kubebuilder 注解可以看到其资源路径为jobs、短名为vcjob/vj且支持status子资源// genclient // kubebuilder:object:roottrue // kubebuilder:resource:pathjobs,shortNamevcjob;vj // kubebuilder:subresource:status // Job defines the volcano job. type Job struct { metav1.TypeMeta json:,inline // optional metav1.ObjectMeta json:metadata,omitempty protobuf:bytes,1,opt,namemetadata // Specification of the desired behavior of the volcano job, including the minAvailable // optional Spec JobSpec json:spec,omitempty protobuf:bytes,2,opt,namespec // Current status of the volcano Job // optional Status JobStatus json:status,omitempty protobuf:bytes,3,opt,namestatus }JobList则是标准的列表对象job.go// kubebuilder:object:roottrue // JobList defines the list of jobs. type JobList struct { metav1.TypeMeta json:,inline metav1.ListMeta json:metadata,omitempty protobuf:bytes,1,opt,namemetadata Items []Job json:items protobuf:bytes,2,rep,nameitems }下文按设计文档的脉络逐一展开JobSpec的核心能力。多 Pod 模板spec.tasks高性能计算任务通常包含不同类型的角色例如 TensorFlowps/worker、Sparkdriver/executor。为此Job引入tasks字段设计文档早期写作taskSpecs当前 API 的 json tag 为tasks支持在同一个 Job 内声明多个 Pod 模板// TaskSpec specifies the task specification of Job. type TaskSpec struct { // Name specifies the name of tasks Name string json:name,omitempty protobuf:bytes,1,opt,namename // Replicas specifies the replicas of this TaskSpec in Job Replicas int32 json:replicas,omitempty protobuf:bytes,2,opt,namereplicas // Specifies the pod that will be created for this TaskSpec // when executing a Job Template v1.PodTemplateSpec json:template,omitempty protobuf:bytes,4,opt,nametemplate // Specifies the lifecycle of task // optional Policies []LifecyclePolicy json:policies,omitempty protobuf:bytes,5,opt,namepolicies }JobController依据spec.tasks中每个任务的template与replicas创建 Pod并将所创建 Pod 的OwnerReference指向该Job即 Pod 由 Job 控制、随 Job 回收。一个多任务模板的示例如下一个 2 ps 5 worker 的 TensorFlow 作业apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: tasks: - name: ps replicas: 2 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img从当前源码看TaskSpec 在原始设计之上又扩展了若干字段均可选MinAvailable *int32任务级的最小可用副本数默认等于该任务的ReplicasTopologyPolicyNUMA 拓扑策略枚举值none / best-effort / restricted / single-numa-nodeMaxRetry int32任务级最大重试次数默认 3DependsOn *DependsOn任务间依赖见下文交互章节PartitionPolicy将任务副本划分为若干子组的分区策略需满足Replicas TotalPartitions * PartitionSize。这些扩展对应了 Gang 感知驱逐、任务启动顺序等后续设计文档如 gang-aware-eviction-design.md但name replicas template policies四元组仍是 API 的骨架。Job 输入/输出spec.volumes大多数高性能负载都会处理输入/输出数据。Volcano 用VolumeSpec以声明式方式描述 Job 的数据卷JobController负责把卷挂载进每个 Task 的 Pod// VolumeSpec defines the specification of Volume, e.g. PVC. type VolumeSpec struct { // Path within the container at which the volume should be mounted. Must not contain :. MountPath string json:mountPath protobuf:bytes,1,opt,namemountPath // defined the PVC name // optional VolumeClaimName string json:volumeClaimName,omitempty protobuf:bytes,2,opt,namevolumeClaimName // VolumeClaim defines the PVC used by the VolumeMount. // optional VolumeClaim *v1.PersistentVolumeClaimSpec json:volumeClaim,omitempty protobuf:bytes,3,opt,namevolumeClaim }在JobSpec中通过Volumes []VolumeSpec引用。语义规则源自设计文档spec.volumes为nil时表示用户自行管理数据Job 不做任何卷操作若某个VolumeSpec的volumeClaim内联 PVC 规格与volumeClaimName已有 PVC 名称都为空或对应 PVC 不存在则对每个 Task/Pod 使用emptyDir卷兜底。当前 VolumeSpec 实现还增加了 kubebuilder 校验mountPath必填且不允许包含:volumeClaimName长度上限 253。另外从 admit_job.go 的更新校验逻辑可以看出一个实现细节当volumeClaim内联给出时控制器会回填 PVC 名称并清空volumeClaimName后再做一致性比较——即内联 PVC 的名称由 Job 控制器维护而非用户手工同步。状态与阶段Phases 和 ConditionsJob 的当前状态由status.state承载phase给出生命周期的高层摘要reason与message提供最近一次阶段迁移的原因与人类可读详情。当前实现还附带了lastTransitionTimejob.go// JobState contains details for the current state of the job. type JobState struct { // The phase of Job. // optional Phase JobPhase json:phase,omitempty protobuf:bytes,1,opt,namephase // Unique, one-word, CamelCase reason for the phases last transition. // optional Reason string json:reason,omitempty protobuf:bytes,2,opt,namereason // Human-readable message indicating details about last transition. // optional Message string json:message,omitempty protobuf:bytes,3,opt,namemessage // Last time the condition transit from one phase to another. // optional LastTransitionTime metav1.Time json:lastTransitionTime,omitempty protobuf:bytes,4,opt,namelastTransitionTime }设计文档定义的阶段枚举JobPhase及允许的迁移关系如下。表格中空单元格表示不允许迁移到该目标阶段From \ ToPendingAbortedRunningCompletedTerminatedPending***Aborted**Running****Completed*Terminated*Restarting、Aborting、Terminating是临时态用于避免竞态例如TerminateJobAction触发后会相继收到多个PodEvictedEvent如果 Job 没有先进入Terminating这类中间态这些事件会被重复处理。当前源码中的完整阶段集合job.go在设计文档基础上补充了Completing任务满足完成条件、开始清理与Failed重启次数达到maxRetry上限const ( // Pending is the phase that job is pending in the queue, waiting for scheduling decision Pending JobPhase Pending // Aborting is the phase that job is aborted, waiting for releasing pods Aborting JobPhase Aborting // Aborted is the phase that job is aborted by user or error handling Aborted JobPhase Aborted // Running is the phase that minimal available tasks of Job are running Running JobPhase Running // Restarting is the phase that the Job is restarted, waiting for pod releasing and recreating Restarting JobPhase Restarting // Completing is the phase that required tasks of job are completed, job starts to clean up Completing JobPhase Completing // Completed is the phase that all tasks of Job are completed Completed JobPhase Completed // Terminating is the phase that the Job is terminated, waiting for releasing pods Terminating JobPhase Terminating // Terminated is the phase that the job is finished unexpected, e.g. events Terminated JobPhase Terminated // Failed is the phase that the job is restarted failed reached the maximum number of retries. Failed JobPhase Failed )从源码结构看阶段并非自由迁移而是由 Job 控制器中的状态机严格驱动factory.go 的NewState函数按当前phase分派到对应的状态处理器每个状态目录一一对应 pkg/controllers/job/state/ 下的实现文件// NewState gets the state from the volcano job Phase. func NewState(jobInfo *apis.JobInfo) State { job : jobInfo.Job switch job.Status.State.Phase { case vcbatch.Pending: return pendingState{job: jobInfo} case vcbatch.Running: return runningState{job: jobInfo} case vcbatch.Restarting: return restartingState{job: jobInfo} case vcbatch.Terminated, vcbatch.Completed, vcbatch.Failed: return finishedState{job: jobInfo} case vcbatch.Terminating: return terminatingState{job: jobInfo} case vcbatch.Aborting: return abortingState{job: jobInfo} case vcbatch.Aborted: return abortedState{job: jobInfo} case vcbatch.Completing: return completingState{job: jobInfo} } // Its pending by default. return pendingState{job: jobInfo} }State接口统一暴露Execute(act Action)方法各状态内部只接受符合迁移表的ActionAbortJob、RestartJob、TerminateJob、ResumeJob、CompleteJob、SyncJob等从而在代码层面保证了设计文档中阶段迁移矩阵的约束。未识别的阶段默认按Pending处理。JobStatus除state外还提供各阶段 Pod 计数。当前实现job.go的完整字段为字段含义state当前JobStatephase reason message lastTransitionTimeminAvailable该 Job 实际生效的最小可用 Pod 数taskStatusCount按任务名统计的各 Pod 阶段计数map: taskName → phase 计数pending/running/succeeded/failed各阶段 Pod 总数terminating/unknownTerminating / Unknown 阶段 Pod 数version作业当前版本重启递增retryCountJob 累计重试次数配合maxRetryrunningDuration从 Running 到完成的时长controlledResourcesJob 控制的其他资源如 Service、ConfigMap清单conditions导致当前状态的JobCondition列表含status与lastTransitionTime错误处理LifecyclePolicyJob 创建后会发生多种事件Pod 成功、Pod 失败、Pod 被驱逐等其中部分事件对 Job 是致命的如 MPI Job 中任一 Pod 失败。为此引入LifecyclePolicy以事件 → 动作的形式让用户按框架语义配置容错行为。事件Event枚举当前实现在 events.goconst ( // AllEvents means all event AllEvents Event * // PodFailedEvent is triggered if Pod was failed PodFailedEvent Event PodFailed // PodEvictedEvent is triggered if Pod was deleted PodEvictedEvent Event PodEvicted // JobUnknownEvent is triggered when part of pods cant be scheduled // while some are already running in gang-scheduling case JobUnknownEvent Event Unknown // OutOfSyncEvent is triggered if Pod/Job were updated OutOfSyncEvent Event OutOfSync // CommandIssuedEvent is triggered if a command is raised by user CommandIssuedEvent Event CommandIssued // TaskCompletedEvent is triggered if the Replicas amount of pods in one task are succeed TaskCompletedEvent Event TaskCompleted )动作Action枚举当前实现在 actions.goconst ( // AbortJobAction: all Pod of Job will be evicted, and no Pod will be recreated // (the job can be resumed later via ResumeJob) AbortJobAction Action AbortJob // RestartJobAction: the whole job will be restarted RestartJobAction Action RestartJob // TerminateJobAction: the job is terminated and can not be resumed TerminateJobAction Action TerminateJob // CompleteJobAction: unfinished pods will be killed, job completed CompleteJobAction Action CompleteJob // ResumeJobAction: resume an aborted job ResumeJobAction Action ResumeJob // SyncJobAction: sync Job/Pod status SyncJobAction Action SyncJob )策略结构体LifecyclePolicyjob.go// LifecyclePolicy specifies the lifecycle and error handling of task and job. type LifecyclePolicy struct { // The action that will be taken to the PodGroup according to Event. Action v1alpha1.Action json:action,omitempty protobuf:bytes,1,opt,nameaction // The Event recorded by scheduler; the controller takes actions according to this Event. Event v1alpha1.Event json:event,omitempty protobuf:bytes,2,opt,nameevent // The Events recorded by scheduler (multiple events per policy). // optional Events []v1alpha1.Event json:events,omitempty protobuf:bytes,3,opt,nameevents // The exit code of the pod container, controller will take action according to this code. // Note: only one of Event or ExitCode can be specified. // optional ExitCode *int32 json:exitCode,omitempty protobuf:bytes,4,opt,nameexitCode // Timeout is the grace period for controller to take actions. // Default to nil (take action immediately). // optional Timeout *metav1.Duration json:timeout,omitempty protobuf:bytes,5,opt,nametimeout }JobSpec与TaskSpec都包含policies字段二者是默认值 覆盖关系spec.policies是 Job 级默认策略spec.tasks[i].policies会覆盖对应任务的默认策略。示例一机器学习训练任务任一任务失败/被驱逐即重启整个 Job。由于所有任务都不单独设置策略全部任务都继承 Job 级event: * → RestartJobapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: # If any event here, restart the whole job. policies: - event: * action: RestartJob tasks: - name: ps replicas: 1 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img示例二Spark 这类大数据框架的差异化容错。driver 失败则重启整个 Jobexecutor 失败则仅靠 Pod 自身的restartPolicy: OnFailure原地重启不牵连整个作业apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: spark-job spec: tasks: - name: driver replicas: 1 policies: - event: * action: RestartJob template: spec: containers: - name: driver image: driver-img - name: executor replicas: 5 template: spec: containers: - name: executor image: executor-img restartPolicy: OnFailure值得注意的演进点当前实现除了Event还支持按容器退出码ExitCode触发策略以及一条策略关联多个事件Events。此外spec.maxRetry默认 3job.go限制了重启这类动作的总次数超过后 Job 进入Failed阶段——这是设计文档发布后为RestartJob引入的熔断机制。与其他特性的交互Admission 校验设计文档要求准入校验必须保证以下预期行为spec.minAvailable sum(spec.tasks.replicas)spec.tasks数组中无重复的 task 名LifecyclePolicy数组Job 级与 Task 级中无重复的事件处理器。当前实现在 admit_job.go 的validateJobCreate中落地并通过 ValidatingWebhook/jobs/validatewebhook 名validatejob.volcano.sh拦截 Create/Update 请求。除上述三项外实际校验集合还包括未声明任何 task 时直接拒绝No task specified in job spec启用了 MPI 插件时必须能按插件参数找到 master 任务The specified mpi master task was not found生成的 Pod 名job-taskName-index规则必须通过 Kubernetes 限定名长度校验validateK8sPodNameLength每个 task 的template走 Kubernetes 原生ValidatePodTemplate校验spec.plugins中的插件名必须在插件注册表中存在unable to find job plugin引用的spec.queue必须存在且状态为Open且不能提交到root队列或非叶子队列层级队列场景若任务间声明了DependsOn依赖依赖图必须是有向无环图DAG否则拒绝。更新操作的约束validateJobUpdate更为严格不能增删 task除minAvailable、tasks[*].replicas与PriorityClassName之外的 spec 字段不可变更——从源码结构看这正是当前作业扩缩容能力scale up/down所允许的唯一可变面。CoSchedulingGang 调度Gang 调度是 TF、MPI 等负载的刚需要么全体就绪要么全体不启动。spec.minAvailable指明至少多少个 Pod 一起被调度默认值为 sum(spec.tasks.replicas)准入 webhook 校验spec.minAvailable不得超过总副本数否则拒绝创建若spec.minAvailable 总副本数超出的 Pod 会被随机创建如需按序创建应使用任务优先级机制。示例1 ps 5 workerminAvailable: 6即全量 gangapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: tf-job spec: # minAvailable to run job minAvailable: 6 tasks: - name: ps replicas: 1 template: spec: containers: - name: ps image: ps-img - name: worker replicas: 5 template: spec: containers: - name: worker image: worker-img任务内优先级Task Priority within Job除多模板外同一 Job 内不同任务还可以有不同优先级。Volcano 复用 Pod 模板的PriorityClass表达任务优先级。示例Spark 作业 1 driver 5 executordriver 使用更高优先级的master-priminAvailable: 3表示资源不足时调度器保证1 driver 2 executor也能先成 gang 启动apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: spark-job spec: minAvailable: 3 tasks: - name: driver replicas: 1 template: spec: priorityClass: master-pri containers: - name: driver image: driver-img - name: executor replicas: 5 template: spec: containers: - name: executor image: executor-img设计文档特别提示了一个竞态虽然调度器会优先调度高优先级 Pod但不同 kubelet 之间仍存在低优先级 Pod 先启动的可能任务间依赖即当前 API 中的spec.tasks[].dependsOn字段见 DependsOn 定义正是为处理这类竞态而引入的后继能力。作业间资源共享默认spec.minAvailable等于总副本数即全量 gang若显式调小minAvailable超出该值的 Pod 将在作业之间共享资源无需独占提升集群利用率。示例与上一节相同minAvailable: 3时第 4~6 个 executor 属于可共享部分。Job 插件spec.pluginsTensorFlow、MPI、MxNet 等框架作业还需要设置环境变量、任务间通信、免密 SSH 登录等胶水工作。Volcano 提供 Job API 插件让用户聚焦核心业务。设计文档发布时提供了三个内置插件每个插件都有参数未提供时使用默认值env向每个容器注入VK_TASK_INDEX当前实现同时注入VC_TASK_INDEX作为容器身份索引——见 env 插件 及其常量定义 const.gosvc为任务创建 Service 与*.host域名使 Pod 之间可通信ssh配置免密 SSH支持mpirun/mpiexec等命令。插件以map[插件名]参数列表挂在spec.plugins上完整示例MPI 作业master 驱逐后重启整个 JobapiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: mpi-job spec: minAvailable: 2 schedulerName: volcano policies: - event: PodEvicted action: RestartJob plugins: ssh: [] env: [] svc: [] tasks: - replicas: 1 name: mpimaster template: spec: containers: image: mpi-image name: mpimaster - replicas: 2 name: mpiworker template: spec: containers: image: mpi-image name: mpiworker插件的可用性与参数语义有专门的用户指南如何配置插件、SSH 插件、SVC 插件、ENV 插件、MPI 插件。附录JobSpec完整字段速查以当前源码 job.go 为准JobSpec的全部字段及说明如下设计文档列标注该字段是否出自原始 Job API 设计字段类型 / 默认值说明设计文档schedulerNamestring最大 63 字符tasks.template.spec.schedulerName的默认值是minAvailableint32默认各 task 副本之和该 Job 至少需要同时调度的 Pod 数Gang 规模是volumes[]VolumeSpecJob 级输入/输出卷声明mountPath PVC 名称或内联 PVC 规格是tasks[]TaskSpecMinItems1任务列表每项含 name/replicas/template/policies 及扩展字段是policies[]LifecyclePolicyJob 级默认生命周期策略可被 TaskSpec.policies 覆盖是pluginsmap[string][]string插件名 → 参数列表env/svc/ssh/mpi 等是queuestring默认default作业提交的调度队列是maxRetryint32默认 3作业重试上限超限进入Failed是runningEstimate*Duration用户预估的运行时长否ttlSecondsAfterFinished*int32Completed/Failed 后自动删除 Job 的宽限期否priorityClassNamestring作业级优先级类否minSuccess*int32最小 1最小成功 Pod 数完成条件min-success 特性否networkTopology*NetworkTopologySpec网络拓扑约束mode: hard/soft可配最高允许跨层否TaskSpec扩展字段设计文档之外的演进字段说明minAvailable任务级最小可用副本数默认等于该任务replicastopologyPolicyNUMA 策略none/best-effort/restricted/single-numa-nodemaxRetry任务级重试上限默认 3dependsOn依赖的其他任务名列表 iterationany/all构成 DAG 启动顺序partitionPolicy分区策略totalPartitions、partitionSize、minPartitions、嵌套networkTopology一个可直接参考的完整 Job 样例见 example/job.yaml。小结Volcano 的JobAPI 围绕一个核心思想展开把一组逻辑耦合的 Pod提升为一等调度对象。其骨架由四部分组成——多模板tasks表达异构角色、volumes声明式接入数据、minAvailable定义 Gang 边界、policiesLifecyclePolicy把 Pod 级事件翻译为作业级动作。围绕骨架准入 webhook 保证提交的合法性控制器内的状态机state 目录严格执行阶段迁移表内置插件消化框架胶水逻辑。原始设计文档中关于任务间依赖处理 kubelet 竞态的展望也已落地为当前的dependsOn字段并在 webhook 中做 DAG 校验。理解这套 API 与源码的对应关系是编写、排障和扩展 Volcano 批处理负载的基础。【免费下载链接】volcanoA Cloud Native Batch System (Project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/vol/volcano创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考