claude-skills 微调评估实战为 Fine-Tuning Expert 构建完整的大模型评估指标体系【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills导读在claude-skills仓库的fine-tuning-expert技能体系中评估是微调工作流的四大环节之一数据准备 → 方法选择 → 训练 →评估→ 部署。本文基于仓库中的 evaluation-metrics.md 参考文档展开系统讲解微调后大模型的评估方法从困惑度Perplexity、BLEU/ROUGE/BERTScore 等生成指标到分类与信息抽取等任务级指标再到EvaluationSuite评估框架、多模型对比、LLM-as-Judge 评审与标准 Benchmark 套件。读完本文你将拥有一套可复制、可运行的评估工具箱能独立为任意微调模型设计并执行一次完整、可信的评估并基于结果决定是否迭代训练或进入部署。一、为什么评估是微调成功的关键一环微调本身只是把模型调向数据的方向而评估回答的是最核心的问题微调到底有没有变好在 SKILL.md 定义的 Core Workflow 中评估被列为第四个强制环节并给出明确的检查点要求Checkpoint: collect perplexity, task-specific metrics (BLEU/ROUGE), and latency numbers.同时在约束清单Constraints中明确要求MUST DO始终在留出集held-out set上做评估基准对齐未微调的 base modelMUST NOT DO禁止不经过留出集评估与延迟基准测试就直接部署。这意味着一份合格的评估输出至少应包含三部分困惑度语言建模能力、任务指标BLEU/ROUGE 或分类指标和延迟数据部署前关心并且所有指标都应同时报告 base model 与微调模型的对照值用增量delta而不是绝对值说话。fine-tuning-expert技能对输出模板也有硬性要求实现微调时必须提供包含困惑度与任务指标的评估脚本。本文所讲解的代码就是这份必须提供的评估脚本的完整参考实现。二、核心指标一Perplexity困惑度——语言建模能力的最基本度量2.1 原理与直觉困惑度是评估自回归语言模型最基础、最廉价的指标。它来源于交叉熵损失对一段真实文本模型给出的平均负对数似然越低困惑度就越低说明模型对该文本越不意外、预测越准确。数学上PPL exp(平均损失)因此困惑度越低语言建模能力越好困惑度与损失是单调对应的训练中观察到的eval_loss下降本质上就意味着困惑度下降。评估时使用留出集held-out test set而不是训练集否则只能测出过拟合程度。参考文档给出了一个可运行的实现import torch import math from transformers import AutoModelForCausalLM, AutoTokenizer from torch.utils.data import DataLoader from tqdm import tqdm def calculate_perplexity( model, tokenizer, texts: list[str], batch_size: int 8, max_length: int 2048 ) - float: Calculate perplexity on a test set. Lower perplexity better language modeling performance. model.eval() total_loss 0 total_tokens 0 encodings tokenizer( texts, truncationTrue, max_lengthmax_length, paddingTrue, return_tensorspt ) dataset torch.utils.data.TensorDataset( encodings[input_ids], encodings[attention_mask] ) dataloader DataLoader(dataset, batch_sizebatch_size) with torch.no_grad(): for batch in tqdm(dataloader, descCalculating perplexity): input_ids, attention_mask batch input_ids input_ids.to(model.device) attention_mask attention_mask.to(model.device) outputs model( input_idsinput_ids, attention_maskattention_mask, labelsinput_ids ) # Count actual tokens (not padding) num_tokens attention_mask.sum().item() total_loss outputs.loss.item() * num_tokens total_tokens num_tokens avg_loss total_loss / total_tokens perplexity math.exp(avg_loss) return perplexity # Usage # perplexity calculate_perplexity(model, tokenizer, test_texts) # print(fPerplexity: {perplexity:.2f})2.2 关键实现细节这个实现中有三个值得注意的细节按 token 加权求平均损失由于做了 padding同一 batch 内不同样本的有效 token 数不同。代码用attention_mask.sum()统计每个 batch 的真实 token 数将损失按 token 数加权后取均值避免 padding token 稀释损失。这是与直接用outputs.loss.mean()的关键区别能让困惑度计算更准确。labelsinput_ids自回归目标对因果语言模型传入labels即自动完成 token 平移与掩码一行代码就得到了语言建模损失。model.eval()torch.no_grad()评估阶段必须关闭 dropout 与梯度计算否则指标不可复现且浪费显存。2.3 何时使用、何时不能只用它困惑度适合回答模型对领域文本的拟合程度如何——例如领域适应domain adaptation类微调。但参考文档在后面的任务指标章节也隐式点明困惑度只能衡量预测文本的能力不能衡量任务完成质量。一个困惑度很低的模型可能在指令遵循、摘要事实性上表现很差。因此困惑度通常作为过滤信号与任务级指标配合使用——这也正是 SKILL.md 中collect perplexity, task-specific metrics并列要求的由来。三、核心指标二Generation-Based Metrics——BLEU / ROUGE / BERTScore对于生成类任务摘要、翻译、对话回复需要把模型输出与人工参考答案gold reference做自动对比。参考文档提供了一个一站式评估函数同时计算三类最常用的生成指标from evaluate import load import numpy as np def evaluate_generation( model, tokenizer, test_examples: list[dict], max_new_tokens: int 256 ) - dict: Evaluate model generation quality with multiple metrics. Args: test_examples: List of {input: str, reference: str} # Load metrics bleu load(bleu) rouge load(rouge) bertscore load(bertscore) predictions [] references [] model.eval() for example in tqdm(test_examples, descGenerating): inputs tokenizer(example[input], return_tensorspt).to(model.device) with torch.no_grad(): outputs model.generate( **inputs, max_new_tokensmax_new_tokens, do_sampleFalse, # Greedy for reproducibility pad_token_idtokenizer.pad_token_id ) prediction tokenizer.decode(outputs[0], skip_special_tokensTrue) # Remove input from prediction if model includes it prediction prediction[len(example[input]):].strip() predictions.append(prediction) references.append(example[reference]) # Calculate metrics results {} # BLEU (0-100, higher is better) bleu_result bleu.compute(predictionspredictions, references[[r] for r in references]) results[bleu] bleu_result[bleu] * 100 # ROUGE (0-1, higher is better) rouge_result rouge.compute(predictionspredictions, referencesreferences) results[rouge1] rouge_result[rouge1] results[rouge2] rouge_result[rouge2] results[rougeL] rouge_result[rougeL] # BERTScore (0-1, higher is better) bertscore_result bertscore.compute( predictionspredictions, referencesreferences, langen ) results[bertscore_f1] np.mean(bertscore_result[f1]) return results # Example # metrics evaluate_generation(model, tokenizer, test_data) # print(fBLEU: {metrics[bleu]:.2f}, ROUGE-L: {metrics[rougeL]:.4f})3.1 三个指标的定位与差异指标度量方式取值范围适用场景与局限BLEUn-gram 精确率precision强调流畅度与词汇覆盖0–100越高越好机器翻译最经典偏重词汇级精确匹配ROUGEn-gram 召回率recall及最长公共子序列ROUGE-1/2/L0–1越高越好摘要任务最常用ROUGE-L用 LCS 衡量整体结构BERTScore基于 BERT 上下文化嵌入的语义相似度precision/recall/F10–1越高越好弥补 n-gram 无法捕捉同义词改写的问题代码中的两个细节值得说明references[[r] for r in references]HuggingFaceevaluate库的 BLEU 要求每个预测对应一个 reference 列表支持多参考答案因此需要加一层包裹ROUGE 则接受扁平列表。这类 API 差异是写评估代码最容易踩的坑。do_sampleFalse贪心解码评估必须可复现因此关闭采样、固定解码策略。若需测试不同解码超参temperature、top_p对质量的影响应显式作为实验变量记录而不是让随机性污染评估结果。3.2 注意生成指标 ≠ 事实性n-gram 与语义相似度指标都无法衡量事实是否正确。参考文档在后面的 deployment-optimization.md 关联中反复强调post-deployment evaluation而事实性factuality通常要依赖 LLM-as-Judge 或人工审核——这正是第五节与第八节内容的用武之地。四、核心指标三Task-Specific Metrics——分类与信息抽取当微调目标不是生成文本而是完成结构化任务分类、信息抽取时需要把生成结果解析回标签或实体再用 sklearn 的经典指标度量。参考文档给出两个函数from sklearn.metrics import accuracy_score, f1_score, classification_report import re def evaluate_classification( model, tokenizer, test_examples: list[dict], labels: list[str] ) - dict: Evaluate fine-tuned model on classification task. Args: test_examples: List of {input: str, label: str} labels: List of valid label strings predictions [] true_labels [] model.eval() for example in tqdm(test_examples, descClassifying): inputs tokenizer(example[input], return_tensorspt).to(model.device) with torch.no_grad(): outputs model.generate( **inputs, max_new_tokens20, do_sampleFalse, pad_token_idtokenizer.pad_token_id ) prediction tokenizer.decode(outputs[0], skip_special_tokensTrue) prediction prediction[len(example[input]):].strip().lower() # Extract label from prediction predicted_label None for label in labels: if label.lower() in prediction: predicted_label label break if predicted_label is None: predicted_label labels[0] # Default to first label predictions.append(predicted_label) true_labels.append(example[label]) return { accuracy: accuracy_score(true_labels, predictions), f1_macro: f1_score(true_labels, predictions, averagemacro), f1_weighted: f1_score(true_labels, predictions, averageweighted), classification_report: classification_report(true_labels, predictions) } def evaluate_extraction( model, tokenizer, test_examples: list[dict] ) - dict: Evaluate information extraction tasks. Args: test_examples: List of {input: str, expected_entities: list[str]} total_precision 0 total_recall 0 total_f1 0 for example in test_examples: inputs tokenizer(example[input], return_tensorspt).to(model.device) with torch.no_grad(): outputs model.generate(**inputs, max_new_tokens256, do_sampleFalse) prediction tokenizer.decode(outputs[0], skip_special_tokensTrue) prediction prediction[len(example[input]):].strip() # Extract entities (customize based on output format) predicted_entities set(re.findall(r\b[A-Z][a-z](?:\s[A-Z][a-z])*\b, prediction)) expected_entities set(example[expected_entities]) # Calculate metrics if len(predicted_entities) 0: precision len(predicted_entities expected_entities) / len(predicted_entities) else: precision 0 if len(expected_entities) 0: recall len(predicted_entities expected_entities) / len(expected_entities) else: recall 1.0 if precision recall 0: f1 2 * precision * recall / (precision recall) else: f1 0 total_precision precision total_recall recall total_f1 f1 n len(test_examples) return { precision: total_precision / n, recall: total_recall / n, f1: total_f1 / n }4.1 分类评估的工程要点子串匹配解析标签LLM 输出往往是情感正面这类带自然语言的句子直接做精确匹配会大量失败。代码遍历labels做不区分大小写的子串匹配失败时默认回落到labels[0]需要在构造评估集时把最常见的类别放在首位或调整该回退策略这是一种务实的容错解析。max_new_tokens20分类输出很短限制生成长度既省显存又避免模型发散。同时报告 accuracy、macro-F1、weighted-F1 和完整 classification_report在类别不均衡时accuracy 会掩盖小类别的劣化macro-F1 能暴露这一问题。4.2 抽取评估精确率 / 召回率 / F1信息抽取采用集合层面的 Jaccard 思想将预测实体与期望实体分别转为集合计算交集比例。三个指标的边界情况处理值得学习模型什么都没预测时precision 0期望集合为空时recall 1.0没有漏检两者交集为空时 F1 0。文中用于实体提取的正则\b[A-Z][a-z](?:\s[A-Z][a-z])*\b匹配首字母大写的英文名词短语只是针对英文输出格式的示例代码注释也明确提示customize based on output format——实际使用时应针对模型约定的输出结构编写对应的解析器这一层解析逻辑往往决定了任务指标可信度的高低。五、评估框架EvaluationSuite——把指标组装成可复用管线把零散指标函数组合成一个可注册、可运行、可保存、可对比的评估套件是让评估工程化的关键。参考文档用 dataclass 实现了一个轻量但完整的EvaluationSuitefrom dataclasses import dataclass, field from typing import Callable, Any import json from datetime import datetime dataclass class EvaluationSuite: Complete evaluation suite for fine-tuned models. name: str metrics: dict[str, Callable] field(default_factorydict) results: dict[str, Any] field(default_factorydict) def add_metric(self, name: str, metric_fn: Callable): Add a metric to the suite. self.metrics[name] metric_fn def run(self, model, tokenizer, test_data: dict) - dict: Run all metrics and return results. self.results { model_name: getattr(model.config, _name_or_path, unknown), timestamp: datetime.now().isoformat(), metrics: {} } for metric_name, metric_fn in self.metrics.items(): print(fRunning {metric_name}...) try: result metric_fn(model, tokenizer, test_data.get(metric_name, test_data)) self.results[metrics][metric_name] result except Exception as e: self.results[metrics][metric_name] {error: str(e)} return self.results def save_results(self, path: str): Save results to JSON file. with open(path, w) as f: json.dump(self.results, f, indent2, defaultstr) def compare_with(self, other_results: dict) - dict: Compare results with another evaluation. comparison {} for metric_name, value in self.results[metrics].items(): if metric_name in other_results.get(metrics, {}): other_value other_results[metrics][metric_name] if isinstance(value, (int, float)) and isinstance(other_value, (int, float)): comparison[metric_name] { current: value, baseline: other_value, delta: value - other_value, delta_pct: ((value - other_value) / other_value * 100) if other_value ! 0 else 0 } return comparison # Usage example def create_evaluation_suite() - EvaluationSuite: suite EvaluationSuite(namefine_tuning_eval) # Add perplexity suite.add_metric(perplexity, lambda m, t, d: calculate_perplexity(m, t, d[texts])) # Add generation metrics suite.add_metric(generation, lambda m, t, d: evaluate_generation(m, t, d[generation])) return suite # Run evaluation # suite create_evaluation_suite() # results suite.run(model, tokenizer, test_data) # suite.save_results(eval_results.json)5.1 设计亮点可插拔指标注册add_metric(name, callable)让指标作为一等对象管理新增一个指标只需一行注册测试数据按指标名从test_data中自动分发test_data.get(metric_name, test_data)。容错运行单个指标抛异常不会中断整个套件而是把错误信息记录到结果中——评估几十个指标时任何一个的数据格式问题都不该让整轮白跑。结果落盘save_results将模型名、时间戳和全部指标写入 JSON满足 SKILL.md 中Document hyperparameters and training config的可复现要求。与基线对比compare_with计算当前结果与 baseline 的delta与delta_pct正是 SKILL.md 要求的Benchmark against the base model与未微调的 base model 对照的直接代码化。六、Model Comparison——多模型横向对比与最优模型选择微调工程中经常需要比较多个候选base model、不同 LoRA rank如 r8、r16、不同方法LoRA vs QLoRA。参考文档的ModelComparison类把这一过程封装成 DataFrame 输出方便直接用 pandas 查看和存档import pandas as pd from typing import Optional class ModelComparison: Compare multiple fine-tuned models. def __init__(self): self.models {} self.results {} def add_model(self, name: str, model, tokenizer, adapter_path: Optional[str] None): Register a model for comparison. self.models[name] { model: model, tokenizer: tokenizer, adapter_path: adapter_path } def evaluate_all(self, test_data: dict, metrics: list[str]) - pd.DataFrame: Evaluate all models and return comparison DataFrame. all_results [] for model_name, model_info in self.models.items(): model model_info[model] tokenizer model_info[tokenizer] model_results {model: model_name} for metric in metrics: if metric perplexity: model_results[perplexity] calculate_perplexity( model, tokenizer, test_data[texts] ) elif metric generation: gen_metrics evaluate_generation( model, tokenizer, test_data[generation] ) model_results.update(gen_metrics) all_results.append(model_results) self.results[model_name] model_results return pd.DataFrame(all_results) def get_best_model(self, metric: str, higher_is_better: bool True) - str: Return name of best performing model for a metric. if not self.results: raise ValueError(No results available. Run evaluate_all first.) values {name: r.get(metric, float(-inf) if higher_is_better else float(inf)) for name, r in self.results.items()} if higher_is_better: return max(values, keyvalues.get) else: return min(values, keyvalues.get) # Usage # comparison ModelComparison() # comparison.add_model(base, base_model, tokenizer) # comparison.add_model(lora_r8, lora_model_r8, tokenizer) # comparison.add_model(lora_r16, lora_model_r16, tokenizer) # df comparison.evaluate_all(test_data, [perplexity, generation]) # print(df)使用要点通过add_model注册任意数量的模型含 adapter 路径信息便于追溯evaluate_all在同一份test_data上运行所有候选保证横向可比性结果以pd.DataFrame返回一行一个模型、一列一个指标可直接print(df)或导出 CSVget_best_model支持越高越好BLEU、ROUGE、BERTScore、Accuracy与越低越好Perplexity两种方向并显式抛出先评估再选最优的守卫错误。实际运用时建议把 base model 也注册进去作为对照组——微调的价值在于相对 base model 的提升而非指标绝对值。这与 SKILL.md 工作流第 4 步Benchmark against the base model完全一致。七、评估指标详解指标选择与结果解读参考表参考文档末尾提供了两张高密度速查表直接决定了评估该看什么、怎么看。以下是完整继承并补充说明的版本。7.1 按任务类型选择指标Task TypePrimary MetricsSecondary MetricsGeneral Fine-TuningPerplexity, LossROUGE, BLEUClassificationAccuracy, F1Precision, RecallGenerationROUGE-L, BERTScoreHuman eval, LLM-as-judgeSummarizationROUGE-1/2/LBERTScore, factualityTranslationBLEU, chrFTER, COMETCodepassk, HumanEvalCodeBLEUChat/AssistantLLM-as-judgeUser preference选择原则主指标必须与任务的成功标准同构。例如摘要任务的主指标是 ROUGE召回式、覆盖要点翻译是 BLEU精确式、逐词对照代码任务则直接看passk可执行通过率对话类任务由于答案空间开放通常以 LLM-as-Judge 和人工偏好为主。表中的Secondary Metrics用于交叉验证主指标的可信度例如 ROUGE 高分但 BERTScore 低往往提示答案套模板但语义不符。7.2 结果解读区间参考MetricPoorAcceptableGoodExcellentPerplexity5020-5010-2010BLEU2020-4040-6060ROUGE-L0.30.3-0.50.5-0.70.7BERTScore F10.70.7-0.850.85-0.920.92Accuracy0.60.6-0.80.8-0.90.9重要前提上表是面向通用场景的经验区间同一指标在不同数据集、不同任务上的绝对水平差异极大例如领域术语密集的摘要集 ROUGE 天然偏低。因此解读时应遵循三条原则纵向优先优先看微调模型相对 base model 的 delta而不是绝对值的Good/Excellent标签横向佐证主指标异常时回看 secondary metrics 定位问题是词汇不匹配、语义漂移还是格式错误结合数据规模评估集过小如不足百条时任何区间的置信度都要打折扣。八、LLM-as-Judge——用强模型评审弱模型当自动指标难以覆盖质量的全部内涵对话助手的helpfulness、事实准确性、连贯性时LLM-as-Judge 是业界通行做法用一个更强的模型按既定标准给生成结果打分。参考文档提供了完整实现from openai import OpenAI import json def llm_judge_evaluation( predictions: list[str], references: list[str], inputs: list[str], judge_model: str gpt-4o, criteria: list[str] None ) - list[dict]: Use LLM as judge to evaluate generation quality. Args: predictions: Model outputs references: Reference/gold outputs inputs: Original inputs judge_model: Model to use as judge criteria: Evaluation criteria (default: helpfulness, accuracy, coherence) if criteria is None: criteria [helpfulness, accuracy, coherence, relevance] client OpenAI() judge_prompt You are an expert evaluator. Rate the following model response on a scale of 1-5 for each criterion. Input: {input} Reference Response: {reference} Model Response: {prediction} Rate the model response on these criteria (1poor, 5excellent): {criteria_list} Return your ratings as JSON: {{criterion_name: score, ...}} Also include a brief explanation for each rating. results [] for input_text, pred, ref in zip(inputs, predictions, references): prompt judge_prompt.format( inputinput_text, referenceref, predictionpred, criteria_list\n.join(f- {c} for c in criteria) ) response client.chat.completions.create( modeljudge_model, messages[{role: user, content: prompt}], temperature0 ) # Parse response try: content response.choices[0].message.content # Extract JSON from response json_match re.search(r\{[^}]\}, content) if json_match: scores json.loads(json_match.group()) else: scores {c: 3 for c in criteria} # Default scores except: scores {c: 3 for c in criteria} results.append({ input: input_text, prediction: pred, reference: ref, scores: scores, raw_response: content }) # Aggregate scores aggregated {c: [] for c in criteria} for r in results: for c in criteria: if c in r[scores]: aggregated[c].append(r[scores][c]) summary {c: sum(scores) / len(scores) if scores else 0 for c, scores in aggregated.items()} return { individual_results: results, summary: summary }8.1 工程要点与边界temperature0评审必须确定性优先采样会引入评审噪声破坏可复现性强制结构化输出提示词要求返回 JSON 评分并附带说明代码用正则提取首个 JSON 对象解析失败时回落到中性分 31–5 分制确保流程不中断保留 raw_response逐条保存评审原始响应便于事后审计评审质量也是检查 judge 是否偏好某些措辞的证据多条 criteria 维度默认helpfulness / accuracy / coherence / relevance可按任务替换为事实性、毒性、安全性等。局限提醒LLM-as-Judge 存在系统性偏差例如偏好更长、更流畅的回答。参考文档在速查表中把 Human eval 与 LLM-as-judge 并列为生成任务的 secondary metrics——对关键发布节点建议用小规模人工标注校准 judge 的打分尺度。九、Benchmark Suites——接入 lm-evaluation-harness 跑标准任务如果希望微调效果能在社区公认的标准任务上可比可以接入lm-evaluation-harness。参考文档封装了run_standard_benchmarksfrom lm_eval import evaluator from lm_eval.models.huggingface import HFLM def run_standard_benchmarks( model, tokenizer, tasks: list[str] None, num_fewshot: int 0 ) - dict: Run standard LLM benchmarks using lm-evaluation-harness. Args: model: HuggingFace model tokenizer: Tokenizer tasks: List of tasks (default: common benchmarks) num_fewshot: Number of few-shot examples if tasks is None: tasks [ hellaswag, # Commonsense reasoning arc_easy, # Science questions arc_challenge, # Harder science questions winogrande, # Commonsense reasoning mmlu, # Multi-task language understanding truthfulqa_mc, # Truthfulness ] # Wrap model for lm-eval lm HFLM(pretrainedmodel, tokenizertokenizer) results evaluator.simple_evaluate( modellm, taskstasks, num_fewshotnum_fewshot, batch_sizeauto ) # Extract key metrics summary {} for task in tasks: if task in results[results]: task_results results[results][task] # Get primary metric (usually accuracy) for key, value in task_results.items(): if acc in key or accuracy in key: summary[task] value break return { full_results: results, summary: summary } # Usage with common benchmarks BENCHMARK_TASKS { reasoning: [hellaswag, winogrande, arc_easy, arc_challenge], knowledge: [mmlu, triviaqa], code: [humaneval, mbpp], math: [gsm8k, math], safety: [truthfulqa_mc, toxigen] }9.1 默认任务集说明了什么默认的六个任务分别覆盖不同能力维度任务考察维度HellaSwag常识推理句子补全ARC Easy / Challenge科学问答区分难易两档WinoGrande指代消解常识推理MMLU多任务语言理解57 个学科TruthfulQA (MC)真实性对抗性误导问题num_fewshot控制 few-shot 示例数直接影响分数可比性对比多个模型时必须使用相同的 few-shot 配置batch_sizeauto让 harness 自动选择 batch兼顾显存与速度BENCHMARK_TASKS常量按 reasoning / knowledge / code / math / safety 分组便于按微调目标挑选子集——例如代码微调只跑 code 组避免在无关任务上浪费算力。标准 benchmark 适合做通用能力的回归检测确认微调没有破坏基础能力而领域任务效果仍需前几节的自定义评估集来判断。十、评估结果如何反哺微调迭代评估不是终点而是微调闭环的决策输入。参考文档在结尾明确列出了与本主题配合的兄弟参考它们共同构成fine-tuning-expert的完整知识链参考文档与评估的关系hyperparameter-tuning.md依据评估结果调整超参——参考其Common Issues表Loss 不下降→用 LR finder 调学习率过拟合→降 epoch、加 dropout评估集上的 eval_loss 是load_best_model_at_end与metric_for_best_model的选择依据dataset-preparation.md评估集从哪来其中的create_stratified_split默认 10% 留出与质量过滤create_quality_filter直接决定评估集质量进而决定评估结论可信度deployment-optimization.md部署前的性能评估其中的benchmark_inference补充延迟/吞吐指标与本文的质量指标共同构成 SKILL.md 要求的quality latency双维度上线门槛一个推荐的迭代工作流与 SKILL.md 的 Core Workflow 对齐用 dataset-preparation.md 中的create_stratified_split固定一份 10% 的留出评估集训练完成后用本文的EvaluationSuite在同一份评估集上评估 base model 与微调模型用ModelComparison对比不同 rank / 学习率 / epoch 的候选用get_best_model选优结合 hyperparameter-tuning.md 的过拟合告警逻辑FineTuningCallback中eval_loss train_loss * 1.5触发告警判断是否需要回炉上线前用 deployment-optimization.md 的benchmark_inference补齐延迟数据质量 延迟双指标达标后再合并 adapter、量化部署。十一、结语构建你自己的评估管线本文完整继承了 evaluation-metrics.md 的全部代码与速查表并补充了每个指标的实现细节、适用边界与仓库内关联证据。一套可靠的评估管线应具备四个特征分层覆盖困惑度语言建模→ 生成指标词汇/语义质量→ 任务指标端到端任务完成度→ LLM-as-Judge / 人工开放质量→ 标准 benchmark通用能力回归可复现贪心解码、固定随机种子、temperature0评审、保存完整结果 JSON有基线任何指标都必须有 base model 对照delta 优先于绝对值有闭环评估结果驱动超参调整见 hyperparameter-tuning.md与部署决策见 deployment-optimization.md。把本文的calculate_perplexity、evaluate_generation、evaluate_classification、evaluate_extraction与EvaluationSuite、ModelComparison、llm_judge_evaluation、run_standard_benchmarks组装进fine-tuning-expert技能的输出模板SKILL.md 要求必须提供的evaluation script reporting perplexity, task-specific metrics, and latency你就拥有了一个与仓库技能体系完全对齐、可投入实战的微调评估工具箱。【免费下载链接】claude-skills67 Specialized Skills for Full-Stack Developers. Transform Claude Code into your expert pair programmer.项目地址: https://gitcode.com/GitHub_Trending/claud/claude-skills创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考