十年匠心定制 · 商业建站与技术教学双线并行 咨询热线:400-886-1026 service@lmnt.cn
ARTICLE DETAIL

资讯详情

深耕网站建设与运营推广的一线实战洞察。

Direct Scoring Evaluation

Direct Scoring Evaluation Direct Scoring Evaluation【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-EngineeringYou are an expert evaluator assessing the quality of an AI-generated response.Your TaskEvaluate the response below against the specified criteria. For each criterion:First, identify specific evidence from the responseThen, determine the appropriate score based on the rubricFinally, provide actionable feedbackImportant GuidelinesBe objective and consistentBase scores on explicit evidence, not assumptionsConsider the original task requirementsAvoid length bias - a shorter, better answer outperforms a longer, weaker oneWhen uncertain between two scores, explain your reasoning then chooseOriginal Prompt/Task{{original_prompt}}{{#if context}}Additional Context{{context}} {{/if}}Response to Evaluate{{response}}Evaluation Criteria{{#each criteria}}{{name}} (Weight: {{weight}}){{description}}{{#if rubric}}Rubric:{{#each rubric}}{{score}}: {{description}} {{/each}} {{/if}} {{/each}}Your EvaluationFor each criterion, provide:Evidence: Specific quotes or observations from the responseScore: Your score according to the rubricJustification: Why this score is appropriateImprovement: Specific suggestion for improvementThen provide:Overall Assessment: Summary of qualityKey Strengths: What the response does wellKey Weaknesses: What needs improvementPriority Improvements: Most impactful changesFormat your response as structured JSON: { scores: [ { criterion: {{name}}, evidence: [quote1, quote2], score: {{score}}, maxScore: {{maxScore}}, justification: ..., improvement: ... } ], overallScore: {{score}}, summary: { assessment: ..., strengths: [..., ...], weaknesses: [..., ...], priorities: [..., ...] } }### 模板的三个关键设计 1. **证据先行Evidence First**模板要求裁判在打分前先引用响应中的具体片段作为证据Evidence: Specific quotes or observations并在 JSON 输出中以 evidence 数组承载。这与 [direct-score.ts](https://link.gitcode.com/i/24346f3ac2ef304a6b33d1240b62139c) 的输出 schema 中的 evidence: z.array(z.string()) 一一对应。强制证据先行能显著抑制凭感觉打分让分数可追溯。 2. **显式偏差约束**Avoid length bias、Base scores on explicit evidence, not assumptions 这两条直接回应了 [llm-evaluator.md](https://link.gitcode.com/i/7642e2c4b50f8e3f19db33af8906afb3) 中记录的冗长偏差与假设推断问题。此外When uncertain between two scores, explain your reasoning then choose 对应仓库建议的 Justification First 实现策略——先解释再给分降低随意性。 3. **结构化 JSON 输出**强制裁判以固定 JSON 结构返回 scores[]、overallScore 与 summary{assessment, strengths, weaknesses, priorities}。这使得下游代码可以直接 JSON.parse 结果并进行加权计算见第四节也天然适合接入 Agent 工具调用链路。 ## 三、变量插槽体系与评分标准Rubric定义 模板通过 Handlebars 风格的 {{变量}} 插槽完成运行时填充官方变量清单如下 | 变量 | 说明 | 是否必填 | |------|------|----------| | original_prompt | 生成该响应的原始提示词/任务 | 是 | | context | 额外上下文RAG 检索文档、对话历史等 | 否 | | response | 被评测的响应文本 | 是 | | criteria | 评测标准数组 | 是 | | criteria.name | 标准名称如 Accuracy | 是 | | criteria.weight | 标准权重 | 是 | | criteria.description | 该标准衡量的具体内容 | 是 | | criteria.rubric | 各分值档位的文字描述 | 否 | ### criteria 与 rubric 的构造示例 模板自带的完整示例输入对应向高中生解释量子纠缠场景 json { original_prompt: Explain quantum entanglement to a high school student, response: Quantum entanglement is like having two magic coins..., criteria: [ { name: Accuracy, weight: 0.4, description: Scientific correctness of the explanation, rubric: [ { score: 1, description: Fundamentally incorrect }, { score: 3, description: Mostly correct with some errors }, { score: 5, description: Completely accurate } ] }, { name: Accessibility, weight: 0.3, description: Understandable for a high school student }, { name: Engagement, weight: 0.3, description: Interesting and memorable } ] }注意Accessibility与Engagement未提供rubric此时裁判将基于通用评分标准打分。若提供了rubric模板会逐档位渲染- **1**: Fundamentally incorrect这样的描述供裁判严格对齐。评分标尺Scale约束仓库在源码层面对评分标尺做了枚举约束。在 direct-score.ts 中RubricSchema定义了scale只能是1-3 | 1-5 | 1-10默认1-5const RubricSchema z.object({ scale: z.enum([1-3, 1-5, 1-10]).default(1-5), levelDescriptions: z.record(z.string(), z.string()).optional() });maxScore由标尺右端解析而来const maxScore parseInt(scale.split(-)[1])direct-score.ts随后被写入每个 criterion 的输出中。同时CriterionSchema对weight做了z.number().min(0).max(1).default(1)的约束保证权重落在合法区间。完整输入 schema 还要求criteria数组至少包含 1 项z.array(CriterionSchema).min(1)从类型层面杜绝空标准评测。四、源码级落地从提示词到可运行的 Direct Score 工具提示词模板只是配方真正让它跑起来的是仓库中基于 Vercel AI SDK 与 Zod 实现的工具链。核心实现在 direct-score.ts。4.1 工具定义与输入输出契约export const DirectScoreInputSchema z.object({ response: z.string().describe(The LLM response to evaluate), prompt: z.string().describe(The original prompt that generated the response), context: z.string().optional().describe(Additional context), criteria: z.array(CriterionSchema).min(1).describe(Evaluation criteria), rubric: RubricSchema.optional() });输出契约DirectScoreOutputSchema与提示词模板要求的 JSON 完全对齐并在其之上补充了两个关键字段weightedScore按权重加权的总分metadata包含evaluationTimeMs、model、criteriaCount用于评测链路的观测与审计。4.2 执行流程内置裁判系统提示词executeDirectScore是核心函数direct-score.ts它将模板的核心思想内嵌为一段精炼的 system promptconst systemPrompt You are an expert evaluator. Assess the response against each criterion. For each criterion: 1. Find specific evidence in the response 2. Score according to the rubric (1-${maxScore} scale) 3. Justify your score 4. Suggest one improvement Be objective and consistent. Base scores on explicit evidence.;随后把原始 prompt、可选 context、被评测响应、criteria含权重与描述以及可选的 rubric 档位描述拼装成 user prompt调用generateText模型来自环境配置temperature: 0.3低温度保证评测稳定性。最后JSON.parse(result.text)解析裁判输出并进入汇总计算。4.3 加权总分与整体分计算逻辑这是评测结果能否被量化的关键一环direct-score.tsconst totalWeight input.criteria.reduce((sum, c) sum c.weight, 0); const weightedSum parsed.scores.reduce((sum, s) { const criterion input.criteria.find(c c.name s.criterion); return sum (s.score * (criterion?.weight || 1)); }, 0); const overallScore parsed.scores.reduce((sum, s) sum s.score, 0) / parsed.scores.length; const weightedScore weightedSum / totalWeight;overallScore所有维度分数的算术平均weightedScore各维度分数乘以其权重后的加权和再除以总权重即Σ(score × weight) / Σweight。由于模板中weight总和通常为 1如 0.40.30.3weightedScore此时就是加权平均。仓库还要求权重必须通过权重之和归一化即便权重之和不为 1 也能得到 0-1 之间的规范化结果。最终两个分数都四舍五入到两位小数返回。实现提示scores数组中的maxScore由标尺动态计算如1-5标尺下maxScore 5因此前端渲染7/10或4/5时无需硬编码。4.4 异常兜底评测过程被try/catch包裹direct-score.ts一旦裁判输出非 JSON、模型调用失败等会返回success: falsescores为空数组、分数归零并在summary.assessment中记录失败原因与耗时。这意味着提示词模板的强制 JSON在实际生产环境必须有容错设计而不仅仅是提示词约束。五、工具描述与 Skill 封装让 Agent 学会调用为了让 Agent 自主决定何时使用该工具仓库在 tools/evaluation/direct-score.md 中给出了带语义描述的版本export const directScore tool({ description: Evaluate a response by scoring it against specific criteria. Use this for objective evaluations where you need to assess quality dimensions like accuracy, completeness, clarity, or task adherence. Returns structured scores with justifications., parameters: z.object({ response: z.string().describe(The LLM response to evaluate), prompt: z.string().describe(The original prompt/instruction that generated the response), context: z.string().optional() .describe(Additional context like retrieved documents or conversation history), criteria: z.array(z.object({ name: z.string().describe(Name of the criterion (e.g., Accuracy)), description: z.string().describe(What this criterion measures), weight: z.number().min(0).max(1).default(1) .describe(Relative importance, weights should sum to 1) })).min(1).describe(Evaluation criteria to score against), rubric: z.object({ scale: z.enum([1-3, 1-5, 1-10]).default(1-5), levelDescriptions: z.record(z.string(), z.string()).optional() .describe(Optional descriptions for each score level) }).optional().describe(Scoring rubric configuration) }), execute: async (input) evaluateWithLLM(input) });关键点description中明确标注适用场景objective evaluations like accuracy, completeness, clarity帮助路由层将客观评测任务导向 direct scoring将主观任务导向 pairwise-comparison-prompt.md每个字段的.describe()注释会作为 LLM 的函数调用 schema 提示让 Agent 知道weight应满足权重总和为 1该工具在 src/tools/evaluation/index.ts 中统一导出并经 src/index.ts 暴露为DirectScoreInput/DirectScoreOutput类型供上层 Agent 与业务代码引用。六、最佳实践把提示词用对的关键约束模板结尾给出了 5 条必须遵守的最佳实践结合源码实现补充如下Evidence First证据先行先收集证据再打分。模板在Your Task中强制了 1→2→3 的顺序实现层面evidence数组是输出 schema 的必填字段从结构上保证先引证、后评分。Rubric Alignment严格对齐评分标准Stick to rubric definitions, dont interpolate——裁判不得在档位之间自行内插分值。这与 tools/evaluation/direct-score.md 的 Implementation Notes 中Calibration: Include few-shot examples of scores at each level在每档位附上 few-shot 示例以校准互为补充。Constructive Feedback建设性反馈improvement字段必须可执行。模板要求输出Priority Improvements: Most impactful changes实现中每个 criterion 都有独立improvement字符串便于直接回流给生成模型迭代。Consistency跨评测一致性同一评测体系内使用相同的标准。实践中建议固定temperature: 0.3见 direct-score.ts并用 llm-evaluator.md 中提到的 Cohens κ / Spearmans ρ 等指标监控裁判自身的一致性。Calibration校准以示例评测作为参照减少裁判的随机漂移。七、端到端运行环境配置与测试验证7.1 环境准备仓库在 env.example 中定义环境变量配置读取逻辑见 src/config/index.tsexport const config { openai: { apiKey: process.env.OPENAI_API_KEY || , model: process.env.OPENAI_MODEL || gpt-4o }, anthropic: { apiKey: process.env.ANTHROPIC_API_KEY || } } as const;复制环境变量示例并填入OPENAI_API_KEY可选覆盖OPENAI_MODEL默认gpt-4o依赖与脚本见 package.json测试框架为 Vitest配置见 vitest.config.ts运行示例npx tsx examples/basic-evaluation.ts运行测试npm test或npx vitest。7.2 最小可运行示例examples/basic-evaluation.ts 展示了完整调用方式import dotenv/config; import { EvaluatorAgent } from ../src/agents/evaluator.js; import { validateConfig } from ../src/config/index.js; const agent new EvaluatorAgent(); const result await agent.score({ response: ...机器学习定义与三大类型的说明..., prompt: Explain what machine learning is to a beginner, criteria: [ { name: Accuracy, description: Factual correctness of the explanation, weight: 0.4 }, { name: Clarity, description: Easy to understand for a beginner, weight: 0.3 }, { name: Completeness, description: Covers the key concepts adequately, weight: 0.3 } ], rubric: { scale: 1-5, levelDescriptions: { 1: Poor - Major issues, 2: Below Average - Several issues, 3: Average - Some issues, 4: Good - Minor issues only, 5: Excellent - No issues } } }); // result.overallScore / result.weightedScore / result.summary...【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-Engineering创作声明:本文部分内容由AI辅助生成(AIGC),仅供参考
返回列表