claude-skills Fine-Tuning Expert 超参数调优实战指南学习率、批大小与调度器的系统化配置方案【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills导读超参数选择直接决定大模型微调的成败学习率过高会导致 loss 震荡发散过低则收敛缓慢批大小与显存不匹配会直接 OOM调度器选错则浪费整个训练周期。本文以 claude-skills 仓库中 fine-tuning-expert 技能体系下的 hyperparameter-tuning.md 为骨架系统讲解面向 LLM 微调的学习率、批大小、调度器与优化器的选型逻辑并给出 LR range test、有效批大小估算、Optuna 超参搜索等可直接运行的代码方案。读完本文你将能够针对 Full Fine-Tuning / LoRA / QLoRA 三类微调方法快速确定合理起点并用可复现的脚本完成从配置生成、搜索到监控的完整闭环。为什么超参数是微调成败的分水岭fine-tuning-expert的 SKILL.md 将微调流程划分为五个环节数据集准备 → 方法选择 → 训练 → 评估 → 部署其中训练环节的核心约束是配置超参数、监控 loss 曲线、定期保存 checkpoint并把验证 loss 必须下降、平台或上升代表过拟合作为关键检查点。这说明超参数配置不是孤立一步它同时受上游数据规模与下游评估结果的制约。hyperparameter-tuning.md的开篇即点明主旨超参数选择对微调成功有决定性影响该参考文档为学习率、批大小、调度器和优化策略提供面向 LLM 微调的实用指导。它与其他三份姊妹文档lora-peft.md、dataset-preparation.md、evaluation-metrics.md构成完整的调参知识闭环数据规模决定批大小与 epoch 数PEFT 方法决定学习率量级评估结果反过来驱动下一轮超参数调整。学习率选择不同微调方法各有量级按微调方法划分的学习率基准学习率是最关键的单点超参数。不同微调方法可训练的参数范围差异巨大因此适用量级也截然不同微调方法典型范围推荐起点说明Full Fine-Tuning1e-6 5e-52e-5模型越大学习率越低LoRA1e-5 3e-42e-4可以采用更高学习率QLoRA1e-5 2e-41e-4与 LoRA 接近Prefix Tuning1e-4 1e-23e-4只训练 embedding量级更宽这里的量级差异可以从 PEFT 的原理佐证lora-peft.md 的 Quick Reference 明确指出learning_rate的典型范围是 1e-5 3e-4并强调LoRA 比 full FT 更能承受高学习率因为 LoRA 只更新低秩适配矩阵更新步长对主权重的影响更小。Full Fine-Tuning 之所以要用小一个量级的学习率是因为它直接修改全部权重lora-peft.md 中的显存对比表Llama 3.1 8BFull FT 约 64 GB vs LoRA 约 18 GB vs QLoRA 约 6 GB也提示参数更新规模越大、涉及参数量越多越需要保守的学习率。值得注意的是Prefix Tuning 的可调范围上限高达 1e-2原因在于它只优化少量 soft prompt 嵌入。实践时仍应从推荐起点 3e-4 出发再做调整。用 LR Finder 找到最陡下降学习率与其靠经验盲猜不如用 LR range test 实测。文档给出了完整的 LR 扫描实现设定从 1e-7 到 1e-2 的指数递增区间逐步前向/反向传播记录 loss绘制学习率-损失曲线后选取梯度最陡loss 下降最快处对应的学习率作为起点并在 loss 爆炸超过首步 loss 的 10 倍时提前终止。import torch import matplotlib.pyplot as plt from transformers import Trainer, TrainingArguments def find_learning_rate( model, train_dataset, tokenizer, min_lr: float 1e-7, max_lr: float 1e-2, num_steps: int 100 ) - tuple[list[float], list[float]]: Find optimal learning rate using LR range test. Returns: Tuple of (learning_rates, losses) # Create temporary training args with linearly increasing LR training_args TrainingArguments( output_dir./lr_finder, max_stepsnum_steps, per_device_train_batch_size4, gradient_accumulation_steps4, learning_ratemax_lr, warmup_steps0, logging_steps1, save_strategyno, report_tonone ) trainer Trainer( modelmodel, argstraining_args, train_datasettrain_dataset, tokenizertokenizer ) # Custom LR schedule that increases exponentially lrs [] losses [] multiplier (max_lr / min_lr) ** (1 / num_steps) current_lr min_lr for step in range(num_steps): # Set LR for param_group in trainer.optimizer.param_groups: param_group[lr] current_lr # Training step loss trainer.training_step(model, next(iter(trainer.get_train_dataloader()))) lrs.append(current_lr) losses.append(loss.item()) current_lr * multiplier # Stop if loss explodes if loss.item() losses[0] * 10: break return lrs, losses绘制曲线时文档用相邻步的 loss 差值除以 LR 差值得到离散梯度取梯度最小值下降最陡对应的学习率作为推荐值用红色虚线标注在图上def plot_lr_finder(lrs: list[float], losses: list[float]): Plot learning rate finder results. plt.figure(figsize(10, 6)) plt.semilogx(lrs, losses) plt.xlabel(Learning Rate) plt.ylabel(Loss) plt.title(Learning Rate Finder) # Find suggested LR (steepest descent) gradients [(losses[i1] - losses[i]) / (lrs[i1] - lrs[i]) for i in range(len(losses) - 1)] suggested_idx gradients.index(min(gradients)) suggested_lr lrs[suggested_idx] plt.axvline(xsuggested_lr, colorr, linestyle--, labelfSuggested LR: {suggested_lr:.2e}) plt.legend() plt.savefig(lr_finder.png) print(fSuggested learning rate: {suggested_lr:.2e}) return suggested_lr执行要点扫描应在完整训练的一小步上进行max_steps100左右即可batch size 用正常值示例为 4accumulation 为 4warmup_steps0避免预热干扰扫描结果save_strategyno避免扫描期间产生 checkpoint 开销。批大小优化有效批大小与显存预算的计算有效批大小 单设备批大小 × 梯度累积步数批大小直接影响梯度估计的噪声与收敛稳定性。文档强调有效批大小effective batch size的概念它由per_device_train_batch_size与gradient_accumulation_steps相乘得到。当显存不足以容纳目标批大小时就通过梯度累积在数学上等价格式化更大的批。文档给出的估算函数基于三类粗糙的显存启发式每十亿参数在不同方法下占用的显存不同——Full Fine-Tuning 约 20 GB/Bbf16 权重 优化器状态 梯度LoRA 约 4 GB/Bbf16 推理 少量可训练参数QLoRA 约 1.5 GB/B4-bit 量化 少量可训练参数。这些系数在 lora-peft.md 的显存对比表中可以交叉印证。def calculate_training_config( target_batch_size: int, gpu_memory_gb: float, model_size_b: float, sequence_length: int 2048, method: str qlora ) - dict: Calculate optimal batch size and gradient accumulation. Args: target_batch_size: Desired effective batch size gpu_memory_gb: Available GPU memory model_size_b: Model size in billions sequence_length: Maximum sequence length method: full, lora, or qlora # Memory estimation (rough heuristics) memory_per_param { full: 20, # bf16 params optimizer states gradients lora: 4, # bf16 inference small trainable qlora: 1.5 # 4-bit small trainable } base_memory_gb model_size_b * memory_per_param[method] available_for_batch gpu_memory_gb - base_memory_gb # Memory per sample (rough estimate) tokens_per_gb 1000 * (8 / model_size_b) # Rough scaling max_samples_in_memory int(available_for_batch * tokens_per_gb / sequence_length) max_batch_per_device max(1, max_samples_in_memory) # Calculate gradient accumulation gradient_accumulation max(1, target_batch_size // max_batch_per_device) actual_batch_per_device min(max_batch_per_device, target_batch_size // gradient_accumulation) effective_batch_size actual_batch_per_device * gradient_accumulation return { per_device_train_batch_size: actual_batch_per_device, gradient_accumulation_steps: gradient_accumulation, effective_batch_size: effective_batch_size, estimated_memory_gb: base_memory_gb (actual_batch_per_device * sequence_length / tokens_per_gb) }文档给出的示例输入为target_batch_size32、gpu_memory_gb24如 RTX 4090、model_size_b8如 Llama 3.1 8B、methodqlora输出为per_device_train_batch_size4、gradient_accumulation_steps8、effective_batch_size32即用 4×8 的组合在 24 GB 显存上达到目标有效批大小 32。需要说明的是这些系数是文档作者给出的粗糙启发式实际显存占用会随序列长度、注意力实现方式如是否开启 Flash Attention见 lora-peft.md 中的attn_implementationflash_attention_2、梯度检查点等显著变化生产环境应以实测显存占用为准把估算值当作起点而非结论。按数据集规模的批大小基准批大小还应与数据规模匹配文档给出如下指南数据集规模推荐批大小说明 1,0008-16小批大小以获取更多参数更新1,000 - 10,00016-32标准批大小10,000 - 100,00032-64更大批大小保证稳定性 100,00064-128可以使用更大的批这一表格与 dataset-preparation.md 的数据集规模指南形成联动例如指令跟随任务建议 1,000 起步、5,00010,000 为推荐规模正好落入 16-32 的标称批大小区间。小数据集的过拟合风险可通过更多参数更新来对冲增加 epoch、减小批大小、提高权重衰减这也是文档中 Small Dataset 配置里weight_decay0.05、max_grad_norm0.3等强正则化项的由来。学习率调度器预热 衰减的完整实现统一调度器工厂函数调度器决定学习率在整个训练过程中的演化轨迹。文档提供了一个基于transformers.get_scheduler的统一工厂函数支持cosine、cosine_with_min_lr、constant_with_warmup以及任意get_scheduler内置类型默认预热比例为 3%warmup_ratio0.03并允许为 cosine 指定最低学习率比例默认 10%from transformers import get_scheduler import torch def create_scheduler( optimizer, scheduler_type: str, num_training_steps: int, warmup_ratio: float 0.03, min_lr_ratio: float 0.1 ): Create learning rate scheduler. Args: scheduler_type: cosine, linear, constant_with_warmup, cosine_with_restarts num_training_steps: Total training steps warmup_ratio: Fraction of steps for warmup min_lr_ratio: Minimum LR as fraction of max (for cosine) num_warmup_steps int(num_training_steps * warmup_ratio) if scheduler_type cosine: scheduler get_scheduler( cosine, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) elif scheduler_type cosine_with_min_lr: # Custom cosine with minimum LR from torch.optim.lr_scheduler import CosineAnnealingLR, SequentialLR, LinearLR warmup LinearLR( optimizer, start_factor0.01, end_factor1.0, total_itersnum_warmup_steps ) cosine CosineAnnealingLR( optimizer, T_maxnum_training_steps - num_warmup_steps, eta_minoptimizer.defaults[lr] * min_lr_ratio ) scheduler SequentialLR( optimizer, schedulers[warmup, cosine], milestones[num_warmup_steps] ) elif scheduler_type constant_with_warmup: scheduler get_scheduler( constant_with_warmup, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) else: scheduler get_scheduler( scheduler_type, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) return scheduler调度器选型要点文档原文归纳cosine适合大多数微调任务平滑衰减是最常用的默认选择linear适合短训练任务constant_with_warmup适合极短微调或学习率本身已接近最优的情形cosine_with_restarts适合长训练周期通过周期性重启保留探索能力。其中 warmup预热几乎是强制项SKILL.md 的 MUST DO 约束明确要求Always include a learning rate warmup其作用是避免训练初期在随机初始化的高梯度下直接使用大学习率而导致 loss 尖峰。可视化对比调度器行为文档还提供了可视化脚本用相同基学习率 2e-4、1000 步、3% 预热绘制 cosine、linear、constant_with_warmup 三条学习率曲线便于直观选择def visualize_schedulers(num_steps: int 1000, warmup_ratio: float 0.03): Plot different scheduler behaviors. import matplotlib.pyplot as plt schedulers_to_plot [cosine, linear, constant_with_warmup] base_lr 2e-4 plt.figure(figsize(12, 6)) for sched_type in schedulers_to_plot: # Create dummy optimizer dummy_param torch.nn.Parameter(torch.zeros(1)) optimizer torch.optim.AdamW([dummy_param], lrbase_lr) scheduler create_scheduler( optimizer, scheduler_typesched_type, num_training_stepsnum_steps, warmup_ratiowarmup_ratio ) lrs [] for _ in range(num_steps): lrs.append(optimizer.param_groups[0][lr]) scheduler.step() plt.plot(lrs, labelsched_type) plt.xlabel(Step) plt.ylabel(Learning Rate) plt.title(Learning Rate Schedulers) plt.legend() plt.savefig(schedulers.png)这个脚本只需一个单元素参数即可运行无需加载真实模型适合在任何微调任务开始前快速比对曲线形态。完整训练配置从数据类到 TrainingArguments 的落地方案FineTuningConfig 数据类文档将全部关键超参数收敛到一个 dataclass 中覆盖模型、LoRA、训练、调度、优化、硬件与评估七个维度便于统一管理与版本记录from transformers import TrainingArguments from dataclasses import dataclass from typing import Optional dataclass class FineTuningConfig: Complete fine-tuning configuration. # Model model_name: str method: str qlora # full, lora, qlora # LoRA specific lora_r: int 16 lora_alpha: int 32 lora_dropout: float 0.05 # Training learning_rate: float 2e-4 num_epochs: int 3 batch_size: int 32 max_seq_length: int 2048 # Scheduler scheduler_type: str cosine warmup_ratio: float 0.03 # Optimization weight_decay: float 0.01 max_grad_norm: float 1.0 adam_beta1: float 0.9 adam_beta2: float 0.999 adam_epsilon: float 1e-8 # Hardware gradient_checkpointing: bool True bf16: bool True tf32: bool True # Evaluation eval_steps: int 100 save_steps: int 100 logging_steps: int 10这些默认值与 SKILL.md 中的 Minimal Working Example 保持一致的推荐起点口径LoRA 配置 r16、alpha32约为 rank 的 2 倍、dropout0.05学习率 2e-4cosine 调度3% 预热。其中lora_r与lora_alpha的选型规则在 lora-peft.md 中有更细的指导rank 通常 464小数据集1,000 条应把 rank 减半如 16→8以降低过拟合超大数据集50,000 条可翻倍如 16→32超过 30B 的大模型则再次减半。从配置生成 TrainingArgumentscreate_training_args将上面的数据类与前面实现的批大小估算结合先根据目标批大小、显存、模型规模与方法自动算出单设备批大小与梯度累积步数再组装成TrainingArgumentsdef create_training_args( config: FineTuningConfig, output_dir: str, gpu_memory_gb: float ) - TrainingArguments: Create TrainingArguments from config. # Calculate batch configuration batch_config calculate_training_config( target_batch_sizeconfig.batch_size, gpu_memory_gbgpu_memory_gb, model_size_b8, # Estimate or pass as parameter sequence_lengthconfig.max_seq_length, methodconfig.method ) return TrainingArguments( output_diroutput_dir, num_train_epochsconfig.num_epochs, # Batch size per_device_train_batch_sizebatch_config[per_device_train_batch_size], per_device_eval_batch_sizebatch_config[per_device_train_batch_size], gradient_accumulation_stepsbatch_config[gradient_accumulation_steps], # Learning rate learning_rateconfig.learning_rate, lr_scheduler_typeconfig.scheduler_type, warmup_ratioconfig.warmup_ratio, # Optimization weight_decayconfig.weight_decay, max_grad_normconfig.max_grad_norm, adam_beta1config.adam_beta1, adam_beta2config.adam_beta2, adam_epsilonconfig.adam_epsilon, optimpaged_adamw_8bit if config.method qlora else adamw_torch, # Hardware gradient_checkpointingconfig.gradient_checkpointing, gradient_checkpointing_kwargs{use_reentrant: False}, bf16config.bf16, tf32config.tf32, # Evaluation and saving eval_strategysteps, eval_stepsconfig.eval_steps, save_strategysteps, save_stepsconfig.save_steps, logging_stepsconfig.logging_steps, save_total_limit3, load_best_model_at_endTrue, metric_for_best_modeleval_loss, greater_is_betterFalse, # Misc group_by_lengthTrue, report_to[wandb], run_namef{config.model_name.split(/)[-1]}-{config.method} )几个关键参数的设计意图值得展开optimpaged_adamw_8bitQLoRA 场景下使用 8-bit 分页优化器把优化器状态换页到 CPU显著降低显存占用与 4-bit 量化叠加正是 QLoRA 的显存优势来源gradient_checkpointing_kwargs{use_reentrant: False}使用非 reentrant 的梯度检查点实现避免 PyTorch 新旧版本之间的兼容告警牺牲部分计算换显存group_by_lengthTrue按序列长度分组、长度相近的样本放入同一 batch减少 padding 浪费提升吞吐load_best_model_at_endTruemetric_for_best_modeleval_lossgreater_is_betterFalse训练结束自动回载验证 loss 最低的 checkpoint配合save_total_limit3只保留最近 3 个 checkpoint。用 Optuna 自动化超参数搜索手动调参之外文档提供了基于 Optuna Trainer.hyperparameter_search的自动化方案。搜索空间覆盖六个维度学习率对数均匀采样 1e-53e-4、单设备批大小候选 2/4/8、epoch 数15、预热比例00.1、权重衰减00.1、调度器类型cosine/linear/constant_with_warmupfrom typing import Any import optuna from transformers import Trainer def hyperparameter_search( model_init, train_dataset, eval_dataset, tokenizer, n_trials: int 20, direction: str minimize ) - dict[str, Any]: Run hyperparameter search using Optuna. Args: model_init: Function that returns initialized model n_trials: Number of trials to run direction: minimize for loss, maximize for accuracy def hp_space(trial: optuna.Trial) - dict: return { learning_rate: trial.suggest_float(learning_rate, 1e-5, 3e-4, logTrue), per_device_train_batch_size: trial.suggest_categorical( per_device_train_batch_size, [2, 4, 8] ), num_train_epochs: trial.suggest_int(num_train_epochs, 1, 5), warmup_ratio: trial.suggest_float(warmup_ratio, 0.0, 0.1), weight_decay: trial.suggest_float(weight_decay, 0.0, 0.1), lr_scheduler_type: trial.suggest_categorical( lr_scheduler_type, [cosine, linear, constant_with_warmup] ) } training_args TrainingArguments( output_dir./hp_search, evaluation_strategyepoch, save_strategyno, report_tonone ) trainer Trainer( model_initmodel_init, argstraining_args, train_datasettrain_dataset, eval_dataseteval_dataset, tokenizertokenizer ) best_trial trainer.hyperparameter_search( hp_spacehp_space, backendoptuna, n_trialsn_trials, directiondirection ) return best_trial.hyperparameters使用注意model_init必须是返回已初始化模型的可调用对象因为每次 trial 都要重新初始化模型LoRA 场景下建议传入全新加载的get_peft_model结果确保各 trial 独立directionminimize针对 loss 类指标若目标是准确率等越高越好的指标则改为maximize搜索阶段设置evaluation_strategyepoch、save_strategyno、report_tonone避免每轮 trial 都写 checkpoint 和外部日志控制搜索开销建议先用少量数据、少量 trial如 510 轮做粗筛锁定学习率与批大小的区间后再在完整数据上精搜。训练过程监控识别过拟合与 loss 异常自定义 TrainerCallbackhyperparameter-tuning.md提供了一个监控回调在每次日志写入时计算 loss 变化速率loss velocity在每次评估时比较训练 loss 与验证 loss当验证 loss 超过训练 loss 的 1.5 倍时发出过拟合警告from transformers import TrainerCallback import wandb class FineTuningCallback(TrainerCallback): Custom callback for fine-tuning monitoring. def on_log(self, args, state, control, logsNone, **kwargs): if logs is None: return # Calculate additional metrics if loss in logs and state.global_step 0: # Track loss velocity if hasattr(self, prev_loss): loss_delta logs[loss] - self.prev_loss logs[loss_delta] loss_delta self.prev_loss logs[loss] def on_evaluate(self, args, state, control, metricsNone, **kwargs): if metrics is None: return # Log evaluation metrics train_loss state.log_history[-1].get(loss, 0) if state.log_history else 0 eval_loss metrics.get(eval_loss, 0) # Warn if overfitting if train_loss 0 and eval_loss train_loss * 1.5: print(fWarning: Potential overfitting. Train loss: {train_loss:.4f}, Eval loss: {eval_loss:.4f}) # Add to trainer # trainer.add_callback(FineTuningCallback())监控与评估的联动体现在多个层面evaluation-metrics.md 明确将根据评估结果调整训练作为与本文档的关联点验证 loss 持续不降说明学习率或容量不足验证 loss 回升而训练 loss 持续走低则是过拟合信号应按下文的 Common Issues 表降低 epoch、增大 dropout 或减小学习率训练结束时回载最佳 checkpointload_best_model_at_endTrue依赖eval_loss序列因此评估间隔eval_steps100不宜过大否则最佳 checkpoint 的定位粒度会变粗report_to[wandb]会把日志推送到 Weights Biases便于跨 trial 对比曲线无外部服务时可改为tensorboard或none。三套可直接上手的推荐起点配置文档在 Quick Reference 中给出三套针对不同数据规模与微调方法的起步配置可直接复制进TrainingArguments小数据集1K 条、QLoRA学习率 1e-4、5 个 epoch、单设备批大小 2、梯度累积 8有效批大小 16、cosine 调度、10% 预热、weight_decay0.05、max_grad_norm0.3。高预热比例与强权重衰减都是为了对抗小样本过拟合。TrainingArguments( learning_rate1e-4, num_train_epochs5, per_device_train_batch_size2, gradient_accumulation_steps8, lr_scheduler_typecosine, warmup_ratio0.1, weight_decay0.05, max_grad_norm0.3 )中等数据集1K-10K 条、LoRA学习率 2e-4、3 个 epoch、单设备批大小 4、梯度累积 4有效批大小 16、cosine 调度、3% 预热、weight_decay0.01、max_grad_norm1.0。这是最常见的通用配置也是 SKILL.md Minimal Working Example 使用的口径。TrainingArguments( learning_rate2e-4, num_train_epochs3, per_device_train_batch_size4, gradient_accumulation_steps4, lr_scheduler_typecosine, warmup_ratio0.03, weight_decay0.01, max_grad_norm1.0 )大数据集10K 条、Full Fine-Tuning学习率回落到 2e-5、2 个 epoch、单设备批大小 8、梯度累积 2有效批大小 16、cosine 调度、3% 预热、weight_decay0.01、max_grad_norm1.0。全参数微调用更小学习率 更少 epoch因为数据量大且更新面广。TrainingArguments( learning_rate2e-5, num_train_epochs2, per_device_train_batch_size8, gradient_accumulation_steps2, lr_scheduler_typecosine, warmup_ratio0.03, weight_decay0.01, max_grad_norm1.0 )三套配置的共同规律有效批大小均落在 16 附近、统一使用 cosine 调度与预热说明有效批大小 调度器才是骨架学习率与正则化项才是随场景变化的核心变量。若需在此配置上启动真实训练可直接参考 SKILL.md 中完整的 LoRA SFTTrainer 最小示例与 QLoRA 4-bit 量化变体。常见问题与排查路线文档总结了五类高频问题的定位与解法这也是调参实战中最高频的排错路径问题可能原因解决方案Loss 不下降学习率过低或过高用 LR finder尝试 10x 或 0.1xLoss 尖峰学习率过高、缺少预热增加预热降低学习率过拟合数据集太小、epoch 过多减少 epoch增大 dropout欠拟合学习率过低、LoRA rank 过低提高学习率增大 rankOOM 报错批大小过大减小批大小增大梯度累积其中LoRA rank 过低导致欠拟合与 lora-peft.md 的 rank 选择指南直接呼应rank 决定适配器容量典型范围 464容量不足时模型表达能力受限表现为训练 loss 迟迟压不下来。OOM 则优先按上文calculate_training_config的思路将per_device_train_batch_size降为 1、同步拉高gradient_accumulation_steps如 16必要时再叠加 8-bit 优化器optimpaged_adamw_8bit。与其他参考文档的联动闭环超参数调优不是孤立环节本文档与fine-tuning-expert技能族的其他参考文档构成完整闭环lora-peft.md提供 LoRA/QLoRA 配置、rank 选择、目标模块与适配器合并的细节决定超参数中的lora_r、lora_alpha、lora_dropout与 target modulesdataset-preparation.md数据集规模决定批大小、epoch 数与正则化强度文档末尾明确其与本文档的关联为根据数据集大小调整训练超参数evaluation-metrics.md评估结果验证 loss、perplexity、BLEU/ROUGE 等驱动下一轮超参数调整其 Quick Reference 还给出各指标好坏分界如 perplexity 10 为优秀、BLEU 40-60 为良好deployment-optimization.md超参数确定并完成训练后进入适配器合并与量化部署环节。从源码结构看fine-tuning-expert的职责定位见 SKILL.md 的 frontmatterdomain: contenteditable="false">【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 claude-skills Fine-Tuning Expert 超参数调优实战指南：学习率、批大小与调度器的系统化配置方案
claude-skills Fine-Tuning Expert 超参数调优实战指南学习率、批大小与调度器的系统化配置方案【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills导读超参数选择直接决定大模型微调的成败学习率过高会导致 loss 震荡发散过低则收敛缓慢批大小与显存不匹配会直接 OOM调度器选错则浪费整个训练周期。本文以 claude-skills 仓库中 fine-tuning-expert 技能体系下的 hyperparameter-tuning.md 为骨架系统讲解面向 LLM 微调的学习率、批大小、调度器与优化器的选型逻辑并给出 LR range test、有效批大小估算、Optuna 超参搜索等可直接运行的代码方案。读完本文你将能够针对 Full Fine-Tuning / LoRA / QLoRA 三类微调方法快速确定合理起点并用可复现的脚本完成从配置生成、搜索到监控的完整闭环。为什么超参数是微调成败的分水岭fine-tuning-expert的 SKILL.md 将微调流程划分为五个环节数据集准备 → 方法选择 → 训练 → 评估 → 部署其中训练环节的核心约束是配置超参数、监控 loss 曲线、定期保存 checkpoint并把验证 loss 必须下降、平台或上升代表过拟合作为关键检查点。这说明超参数配置不是孤立一步它同时受上游数据规模与下游评估结果的制约。hyperparameter-tuning.md的开篇即点明主旨超参数选择对微调成功有决定性影响该参考文档为学习率、批大小、调度器和优化策略提供面向 LLM 微调的实用指导。它与其他三份姊妹文档lora-peft.md、dataset-preparation.md、evaluation-metrics.md构成完整的调参知识闭环数据规模决定批大小与 epoch 数PEFT 方法决定学习率量级评估结果反过来驱动下一轮超参数调整。学习率选择不同微调方法各有量级按微调方法划分的学习率基准学习率是最关键的单点超参数。不同微调方法可训练的参数范围差异巨大因此适用量级也截然不同微调方法典型范围推荐起点说明Full Fine-Tuning1e-6 5e-52e-5模型越大学习率越低LoRA1e-5 3e-42e-4可以采用更高学习率QLoRA1e-5 2e-41e-4与 LoRA 接近Prefix Tuning1e-4 1e-23e-4只训练 embedding量级更宽这里的量级差异可以从 PEFT 的原理佐证lora-peft.md 的 Quick Reference 明确指出learning_rate的典型范围是 1e-5 3e-4并强调LoRA 比 full FT 更能承受高学习率因为 LoRA 只更新低秩适配矩阵更新步长对主权重的影响更小。Full Fine-Tuning 之所以要用小一个量级的学习率是因为它直接修改全部权重lora-peft.md 中的显存对比表Llama 3.1 8BFull FT 约 64 GB vs LoRA 约 18 GB vs QLoRA 约 6 GB也提示参数更新规模越大、涉及参数量越多越需要保守的学习率。值得注意的是Prefix Tuning 的可调范围上限高达 1e-2原因在于它只优化少量 soft prompt 嵌入。实践时仍应从推荐起点 3e-4 出发再做调整。用 LR Finder 找到最陡下降学习率与其靠经验盲猜不如用 LR range test 实测。文档给出了完整的 LR 扫描实现设定从 1e-7 到 1e-2 的指数递增区间逐步前向/反向传播记录 loss绘制学习率-损失曲线后选取梯度最陡loss 下降最快处对应的学习率作为起点并在 loss 爆炸超过首步 loss 的 10 倍时提前终止。import torch import matplotlib.pyplot as plt from transformers import Trainer, TrainingArguments def find_learning_rate( model, train_dataset, tokenizer, min_lr: float 1e-7, max_lr: float 1e-2, num_steps: int 100 ) - tuple[list[float], list[float]]: Find optimal learning rate using LR range test. Returns: Tuple of (learning_rates, losses) # Create temporary training args with linearly increasing LR training_args TrainingArguments( output_dir./lr_finder, max_stepsnum_steps, per_device_train_batch_size4, gradient_accumulation_steps4, learning_ratemax_lr, warmup_steps0, logging_steps1, save_strategyno, report_tonone ) trainer Trainer( modelmodel, argstraining_args, train_datasettrain_dataset, tokenizertokenizer ) # Custom LR schedule that increases exponentially lrs [] losses [] multiplier (max_lr / min_lr) ** (1 / num_steps) current_lr min_lr for step in range(num_steps): # Set LR for param_group in trainer.optimizer.param_groups: param_group[lr] current_lr # Training step loss trainer.training_step(model, next(iter(trainer.get_train_dataloader()))) lrs.append(current_lr) losses.append(loss.item()) current_lr * multiplier # Stop if loss explodes if loss.item() losses[0] * 10: break return lrs, losses绘制曲线时文档用相邻步的 loss 差值除以 LR 差值得到离散梯度取梯度最小值下降最陡对应的学习率作为推荐值用红色虚线标注在图上def plot_lr_finder(lrs: list[float], losses: list[float]): Plot learning rate finder results. plt.figure(figsize(10, 6)) plt.semilogx(lrs, losses) plt.xlabel(Learning Rate) plt.ylabel(Loss) plt.title(Learning Rate Finder) # Find suggested LR (steepest descent) gradients [(losses[i1] - losses[i]) / (lrs[i1] - lrs[i]) for i in range(len(losses) - 1)] suggested_idx gradients.index(min(gradients)) suggested_lr lrs[suggested_idx] plt.axvline(xsuggested_lr, colorr, linestyle--, labelfSuggested LR: {suggested_lr:.2e}) plt.legend() plt.savefig(lr_finder.png) print(fSuggested learning rate: {suggested_lr:.2e}) return suggested_lr执行要点扫描应在完整训练的一小步上进行max_steps100左右即可batch size 用正常值示例为 4accumulation 为 4warmup_steps0避免预热干扰扫描结果save_strategyno避免扫描期间产生 checkpoint 开销。批大小优化有效批大小与显存预算的计算有效批大小 单设备批大小 × 梯度累积步数批大小直接影响梯度估计的噪声与收敛稳定性。文档强调有效批大小effective batch size的概念它由per_device_train_batch_size与gradient_accumulation_steps相乘得到。当显存不足以容纳目标批大小时就通过梯度累积在数学上等价格式化更大的批。文档给出的估算函数基于三类粗糙的显存启发式每十亿参数在不同方法下占用的显存不同——Full Fine-Tuning 约 20 GB/Bbf16 权重 优化器状态 梯度LoRA 约 4 GB/Bbf16 推理 少量可训练参数QLoRA 约 1.5 GB/B4-bit 量化 少量可训练参数。这些系数在 lora-peft.md 的显存对比表中可以交叉印证。def calculate_training_config( target_batch_size: int, gpu_memory_gb: float, model_size_b: float, sequence_length: int 2048, method: str qlora ) - dict: Calculate optimal batch size and gradient accumulation. Args: target_batch_size: Desired effective batch size gpu_memory_gb: Available GPU memory model_size_b: Model size in billions sequence_length: Maximum sequence length method: full, lora, or qlora # Memory estimation (rough heuristics) memory_per_param { full: 20, # bf16 params optimizer states gradients lora: 4, # bf16 inference small trainable qlora: 1.5 # 4-bit small trainable } base_memory_gb model_size_b * memory_per_param[method] available_for_batch gpu_memory_gb - base_memory_gb # Memory per sample (rough estimate) tokens_per_gb 1000 * (8 / model_size_b) # Rough scaling max_samples_in_memory int(available_for_batch * tokens_per_gb / sequence_length) max_batch_per_device max(1, max_samples_in_memory) # Calculate gradient accumulation gradient_accumulation max(1, target_batch_size // max_batch_per_device) actual_batch_per_device min(max_batch_per_device, target_batch_size // gradient_accumulation) effective_batch_size actual_batch_per_device * gradient_accumulation return { per_device_train_batch_size: actual_batch_per_device, gradient_accumulation_steps: gradient_accumulation, effective_batch_size: effective_batch_size, estimated_memory_gb: base_memory_gb (actual_batch_per_device * sequence_length / tokens_per_gb) }文档给出的示例输入为target_batch_size32、gpu_memory_gb24如 RTX 4090、model_size_b8如 Llama 3.1 8B、methodqlora输出为per_device_train_batch_size4、gradient_accumulation_steps8、effective_batch_size32即用 4×8 的组合在 24 GB 显存上达到目标有效批大小 32。需要说明的是这些系数是文档作者给出的粗糙启发式实际显存占用会随序列长度、注意力实现方式如是否开启 Flash Attention见 lora-peft.md 中的attn_implementationflash_attention_2、梯度检查点等显著变化生产环境应以实测显存占用为准把估算值当作起点而非结论。按数据集规模的批大小基准批大小还应与数据规模匹配文档给出如下指南数据集规模推荐批大小说明 1,0008-16小批大小以获取更多参数更新1,000 - 10,00016-32标准批大小10,000 - 100,00032-64更大批大小保证稳定性 100,00064-128可以使用更大的批这一表格与 dataset-preparation.md 的数据集规模指南形成联动例如指令跟随任务建议 1,000 起步、5,00010,000 为推荐规模正好落入 16-32 的标称批大小区间。小数据集的过拟合风险可通过更多参数更新来对冲增加 epoch、减小批大小、提高权重衰减这也是文档中 Small Dataset 配置里weight_decay0.05、max_grad_norm0.3等强正则化项的由来。学习率调度器预热 衰减的完整实现统一调度器工厂函数调度器决定学习率在整个训练过程中的演化轨迹。文档提供了一个基于transformers.get_scheduler的统一工厂函数支持cosine、cosine_with_min_lr、constant_with_warmup以及任意get_scheduler内置类型默认预热比例为 3%warmup_ratio0.03并允许为 cosine 指定最低学习率比例默认 10%from transformers import get_scheduler import torch def create_scheduler( optimizer, scheduler_type: str, num_training_steps: int, warmup_ratio: float 0.03, min_lr_ratio: float 0.1 ): Create learning rate scheduler. Args: scheduler_type: cosine, linear, constant_with_warmup, cosine_with_restarts num_training_steps: Total training steps warmup_ratio: Fraction of steps for warmup min_lr_ratio: Minimum LR as fraction of max (for cosine) num_warmup_steps int(num_training_steps * warmup_ratio) if scheduler_type cosine: scheduler get_scheduler( cosine, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) elif scheduler_type cosine_with_min_lr: # Custom cosine with minimum LR from torch.optim.lr_scheduler import CosineAnnealingLR, SequentialLR, LinearLR warmup LinearLR( optimizer, start_factor0.01, end_factor1.0, total_itersnum_warmup_steps ) cosine CosineAnnealingLR( optimizer, T_maxnum_training_steps - num_warmup_steps, eta_minoptimizer.defaults[lr] * min_lr_ratio ) scheduler SequentialLR( optimizer, schedulers[warmup, cosine], milestones[num_warmup_steps] ) elif scheduler_type constant_with_warmup: scheduler get_scheduler( constant_with_warmup, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) else: scheduler get_scheduler( scheduler_type, optimizeroptimizer, num_warmup_stepsnum_warmup_steps, num_training_stepsnum_training_steps ) return scheduler调度器选型要点文档原文归纳cosine适合大多数微调任务平滑衰减是最常用的默认选择linear适合短训练任务constant_with_warmup适合极短微调或学习率本身已接近最优的情形cosine_with_restarts适合长训练周期通过周期性重启保留探索能力。其中 warmup预热几乎是强制项SKILL.md 的 MUST DO 约束明确要求Always include a learning rate warmup其作用是避免训练初期在随机初始化的高梯度下直接使用大学习率而导致 loss 尖峰。可视化对比调度器行为文档还提供了可视化脚本用相同基学习率 2e-4、1000 步、3% 预热绘制 cosine、linear、constant_with_warmup 三条学习率曲线便于直观选择def visualize_schedulers(num_steps: int 1000, warmup_ratio: float 0.03): Plot different scheduler behaviors. import matplotlib.pyplot as plt schedulers_to_plot [cosine, linear, constant_with_warmup] base_lr 2e-4 plt.figure(figsize(12, 6)) for sched_type in schedulers_to_plot: # Create dummy optimizer dummy_param torch.nn.Parameter(torch.zeros(1)) optimizer torch.optim.AdamW([dummy_param], lrbase_lr) scheduler create_scheduler( optimizer, scheduler_typesched_type, num_training_stepsnum_steps, warmup_ratiowarmup_ratio ) lrs [] for _ in range(num_steps): lrs.append(optimizer.param_groups[0][lr]) scheduler.step() plt.plot(lrs, labelsched_type) plt.xlabel(Step) plt.ylabel(Learning Rate) plt.title(Learning Rate Schedulers) plt.legend() plt.savefig(schedulers.png)这个脚本只需一个单元素参数即可运行无需加载真实模型适合在任何微调任务开始前快速比对曲线形态。完整训练配置从数据类到 TrainingArguments 的落地方案FineTuningConfig 数据类文档将全部关键超参数收敛到一个 dataclass 中覆盖模型、LoRA、训练、调度、优化、硬件与评估七个维度便于统一管理与版本记录from transformers import TrainingArguments from dataclasses import dataclass from typing import Optional dataclass class FineTuningConfig: Complete fine-tuning configuration. # Model model_name: str method: str qlora # full, lora, qlora # LoRA specific lora_r: int 16 lora_alpha: int 32 lora_dropout: float 0.05 # Training learning_rate: float 2e-4 num_epochs: int 3 batch_size: int 32 max_seq_length: int 2048 # Scheduler scheduler_type: str cosine warmup_ratio: float 0.03 # Optimization weight_decay: float 0.01 max_grad_norm: float 1.0 adam_beta1: float 0.9 adam_beta2: float 0.999 adam_epsilon: float 1e-8 # Hardware gradient_checkpointing: bool True bf16: bool True tf32: bool True # Evaluation eval_steps: int 100 save_steps: int 100 logging_steps: int 10这些默认值与 SKILL.md 中的 Minimal Working Example 保持一致的推荐起点口径LoRA 配置 r16、alpha32约为 rank 的 2 倍、dropout0.05学习率 2e-4cosine 调度3% 预热。其中lora_r与lora_alpha的选型规则在 lora-peft.md 中有更细的指导rank 通常 464小数据集1,000 条应把 rank 减半如 16→8以降低过拟合超大数据集50,000 条可翻倍如 16→32超过 30B 的大模型则再次减半。从配置生成 TrainingArgumentscreate_training_args将上面的数据类与前面实现的批大小估算结合先根据目标批大小、显存、模型规模与方法自动算出单设备批大小与梯度累积步数再组装成TrainingArgumentsdef create_training_args( config: FineTuningConfig, output_dir: str, gpu_memory_gb: float ) - TrainingArguments: Create TrainingArguments from config. # Calculate batch configuration batch_config calculate_training_config( target_batch_sizeconfig.batch_size, gpu_memory_gbgpu_memory_gb, model_size_b8, # Estimate or pass as parameter sequence_lengthconfig.max_seq_length, methodconfig.method ) return TrainingArguments( output_diroutput_dir, num_train_epochsconfig.num_epochs, # Batch size per_device_train_batch_sizebatch_config[per_device_train_batch_size], per_device_eval_batch_sizebatch_config[per_device_train_batch_size], gradient_accumulation_stepsbatch_config[gradient_accumulation_steps], # Learning rate learning_rateconfig.learning_rate, lr_scheduler_typeconfig.scheduler_type, warmup_ratioconfig.warmup_ratio, # Optimization weight_decayconfig.weight_decay, max_grad_normconfig.max_grad_norm, adam_beta1config.adam_beta1, adam_beta2config.adam_beta2, adam_epsilonconfig.adam_epsilon, optimpaged_adamw_8bit if config.method qlora else adamw_torch, # Hardware gradient_checkpointingconfig.gradient_checkpointing, gradient_checkpointing_kwargs{use_reentrant: False}, bf16config.bf16, tf32config.tf32, # Evaluation and saving eval_strategysteps, eval_stepsconfig.eval_steps, save_strategysteps, save_stepsconfig.save_steps, logging_stepsconfig.logging_steps, save_total_limit3, load_best_model_at_endTrue, metric_for_best_modeleval_loss, greater_is_betterFalse, # Misc group_by_lengthTrue, report_to[wandb], run_namef{config.model_name.split(/)[-1]}-{config.method} )几个关键参数的设计意图值得展开optimpaged_adamw_8bitQLoRA 场景下使用 8-bit 分页优化器把优化器状态换页到 CPU显著降低显存占用与 4-bit 量化叠加正是 QLoRA 的显存优势来源gradient_checkpointing_kwargs{use_reentrant: False}使用非 reentrant 的梯度检查点实现避免 PyTorch 新旧版本之间的兼容告警牺牲部分计算换显存group_by_lengthTrue按序列长度分组、长度相近的样本放入同一 batch减少 padding 浪费提升吞吐load_best_model_at_endTruemetric_for_best_modeleval_lossgreater_is_betterFalse训练结束自动回载验证 loss 最低的 checkpoint配合save_total_limit3只保留最近 3 个 checkpoint。用 Optuna 自动化超参数搜索手动调参之外文档提供了基于 Optuna Trainer.hyperparameter_search的自动化方案。搜索空间覆盖六个维度学习率对数均匀采样 1e-53e-4、单设备批大小候选 2/4/8、epoch 数15、预热比例00.1、权重衰减00.1、调度器类型cosine/linear/constant_with_warmupfrom typing import Any import optuna from transformers import Trainer def hyperparameter_search( model_init, train_dataset, eval_dataset, tokenizer, n_trials: int 20, direction: str minimize ) - dict[str, Any]: Run hyperparameter search using Optuna. Args: model_init: Function that returns initialized model n_trials: Number of trials to run direction: minimize for loss, maximize for accuracy def hp_space(trial: optuna.Trial) - dict: return { learning_rate: trial.suggest_float(learning_rate, 1e-5, 3e-4, logTrue), per_device_train_batch_size: trial.suggest_categorical( per_device_train_batch_size, [2, 4, 8] ), num_train_epochs: trial.suggest_int(num_train_epochs, 1, 5), warmup_ratio: trial.suggest_float(warmup_ratio, 0.0, 0.1), weight_decay: trial.suggest_float(weight_decay, 0.0, 0.1), lr_scheduler_type: trial.suggest_categorical( lr_scheduler_type, [cosine, linear, constant_with_warmup] ) } training_args TrainingArguments( output_dir./hp_search, evaluation_strategyepoch, save_strategyno, report_tonone ) trainer Trainer( model_initmodel_init, argstraining_args, train_datasettrain_dataset, eval_dataseteval_dataset, tokenizertokenizer ) best_trial trainer.hyperparameter_search( hp_spacehp_space, backendoptuna, n_trialsn_trials, directiondirection ) return best_trial.hyperparameters使用注意model_init必须是返回已初始化模型的可调用对象因为每次 trial 都要重新初始化模型LoRA 场景下建议传入全新加载的get_peft_model结果确保各 trial 独立directionminimize针对 loss 类指标若目标是准确率等越高越好的指标则改为maximize搜索阶段设置evaluation_strategyepoch、save_strategyno、report_tonone避免每轮 trial 都写 checkpoint 和外部日志控制搜索开销建议先用少量数据、少量 trial如 510 轮做粗筛锁定学习率与批大小的区间后再在完整数据上精搜。训练过程监控识别过拟合与 loss 异常自定义 TrainerCallbackhyperparameter-tuning.md提供了一个监控回调在每次日志写入时计算 loss 变化速率loss velocity在每次评估时比较训练 loss 与验证 loss当验证 loss 超过训练 loss 的 1.5 倍时发出过拟合警告from transformers import TrainerCallback import wandb class FineTuningCallback(TrainerCallback): Custom callback for fine-tuning monitoring. def on_log(self, args, state, control, logsNone, **kwargs): if logs is None: return # Calculate additional metrics if loss in logs and state.global_step 0: # Track loss velocity if hasattr(self, prev_loss): loss_delta logs[loss] - self.prev_loss logs[loss_delta] loss_delta self.prev_loss logs[loss] def on_evaluate(self, args, state, control, metricsNone, **kwargs): if metrics is None: return # Log evaluation metrics train_loss state.log_history[-1].get(loss, 0) if state.log_history else 0 eval_loss metrics.get(eval_loss, 0) # Warn if overfitting if train_loss 0 and eval_loss train_loss * 1.5: print(fWarning: Potential overfitting. Train loss: {train_loss:.4f}, Eval loss: {eval_loss:.4f}) # Add to trainer # trainer.add_callback(FineTuningCallback())监控与评估的联动体现在多个层面evaluation-metrics.md 明确将根据评估结果调整训练作为与本文档的关联点验证 loss 持续不降说明学习率或容量不足验证 loss 回升而训练 loss 持续走低则是过拟合信号应按下文的 Common Issues 表降低 epoch、增大 dropout 或减小学习率训练结束时回载最佳 checkpointload_best_model_at_endTrue依赖eval_loss序列因此评估间隔eval_steps100不宜过大否则最佳 checkpoint 的定位粒度会变粗report_to[wandb]会把日志推送到 Weights Biases便于跨 trial 对比曲线无外部服务时可改为tensorboard或none。三套可直接上手的推荐起点配置文档在 Quick Reference 中给出三套针对不同数据规模与微调方法的起步配置可直接复制进TrainingArguments小数据集1K 条、QLoRA学习率 1e-4、5 个 epoch、单设备批大小 2、梯度累积 8有效批大小 16、cosine 调度、10% 预热、weight_decay0.05、max_grad_norm0.3。高预热比例与强权重衰减都是为了对抗小样本过拟合。TrainingArguments( learning_rate1e-4, num_train_epochs5, per_device_train_batch_size2, gradient_accumulation_steps8, lr_scheduler_typecosine, warmup_ratio0.1, weight_decay0.05, max_grad_norm0.3 )中等数据集1K-10K 条、LoRA学习率 2e-4、3 个 epoch、单设备批大小 4、梯度累积 4有效批大小 16、cosine 调度、3% 预热、weight_decay0.01、max_grad_norm1.0。这是最常见的通用配置也是 SKILL.md Minimal Working Example 使用的口径。TrainingArguments( learning_rate2e-4, num_train_epochs3, per_device_train_batch_size4, gradient_accumulation_steps4, lr_scheduler_typecosine, warmup_ratio0.03, weight_decay0.01, max_grad_norm1.0 )大数据集10K 条、Full Fine-Tuning学习率回落到 2e-5、2 个 epoch、单设备批大小 8、梯度累积 2有效批大小 16、cosine 调度、3% 预热、weight_decay0.01、max_grad_norm1.0。全参数微调用更小学习率 更少 epoch因为数据量大且更新面广。TrainingArguments( learning_rate2e-5, num_train_epochs2, per_device_train_batch_size8, gradient_accumulation_steps2, lr_scheduler_typecosine, warmup_ratio0.03, weight_decay0.01, max_grad_norm1.0 )三套配置的共同规律有效批大小均落在 16 附近、统一使用 cosine 调度与预热说明有效批大小 调度器才是骨架学习率与正则化项才是随场景变化的核心变量。若需在此配置上启动真实训练可直接参考 SKILL.md 中完整的 LoRA SFTTrainer 最小示例与 QLoRA 4-bit 量化变体。常见问题与排查路线文档总结了五类高频问题的定位与解法这也是调参实战中最高频的排错路径问题可能原因解决方案Loss 不下降学习率过低或过高用 LR finder尝试 10x 或 0.1xLoss 尖峰学习率过高、缺少预热增加预热降低学习率过拟合数据集太小、epoch 过多减少 epoch增大 dropout欠拟合学习率过低、LoRA rank 过低提高学习率增大 rankOOM 报错批大小过大减小批大小增大梯度累积其中LoRA rank 过低导致欠拟合与 lora-peft.md 的 rank 选择指南直接呼应rank 决定适配器容量典型范围 464容量不足时模型表达能力受限表现为训练 loss 迟迟压不下来。OOM 则优先按上文calculate_training_config的思路将per_device_train_batch_size降为 1、同步拉高gradient_accumulation_steps如 16必要时再叠加 8-bit 优化器optimpaged_adamw_8bit。与其他参考文档的联动闭环超参数调优不是孤立环节本文档与fine-tuning-expert技能族的其他参考文档构成完整闭环lora-peft.md提供 LoRA/QLoRA 配置、rank 选择、目标模块与适配器合并的细节决定超参数中的lora_r、lora_alpha、lora_dropout与 target modulesdataset-preparation.md数据集规模决定批大小、epoch 数与正则化强度文档末尾明确其与本文档的关联为根据数据集大小调整训练超参数evaluation-metrics.md评估结果验证 loss、perplexity、BLEU/ROUGE 等驱动下一轮超参数调整其 Quick Reference 还给出各指标好坏分界如 perplexity 10 为优秀、BLEU 40-60 为良好deployment-optimization.md超参数确定并完成训练后进入适配器合并与量化部署环节。从源码结构看fine-tuning-expert的职责定位见 SKILL.md 的 frontmatterdomain: contenteditable="false">【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考