[基于OpenEvals的自动化评估-05]基于JSON相似度的评估[LLM参与的语义匹配]

基于JSON内容的评估可以说是OpenEvals最重要的部分基于OpenEvals的自动化评估-04:基于JSON相似度的评估对用于提供相关评估器的如下两个工厂函数从无LLM参与的角度进行了介绍。我们着重讨论了这两个函数的aggregator、list_aggregator和list_match_mode参数。其余的参数除了exclude_keys均与LLM-as-a-Judge有关。如果有了LLM加持我们在评估的时候将不再仅限于严格的字符串比较可以进行真正的基于自然语言的语义相似度比较这样才能真正发挥AI的能力。defcreate_json_match_evaluator(*,aggregator:Optional[Literal[average,all]]None,list_aggregator:Literal[average,all]all,rubric:Dict[str,str]{},exclude_keys:list[str][],judge:Optional[Union[ModelClient,BaseChatModel,]]None,model:Optional[str]None,use_reasoning:boolTrue,list_match_mode:Literal[superset,subset,same_elements,ordered]same_elements,)-SimpleEvaluatordefcreate_async_json_match_evaluator(*,aggregator:Optional[Literal[average,all]]None,list_aggregator:Literal[average,all]all,rubric:Dict[str,str]{},exclude_keys:list[str][],judge:Optional[Union[ModelClient,BaseChatModel,]]None,model:Optional[str]None,use_reasoning:boolTrue,list_match_mode:Literal[superset,subset,same_elements,ordered]same_elements,)-SimpleAsyncEvaluator1. 参数说明create_json_match_evaluator和create_async_json_match_evaluator与LLM-as-a-Judge有关的参数由如下几个rubric通过一个字段为JSON的每个字段定义评估规则judge以ModelClient或者BaseChatModel的方式指定连接LLM的客户端model以标准的形式指定作为评估模型的名称use_reasoning是否开启推理并使用推理文本来填充EvaluatorResult的commment字段。除了用来定义评估规则的rubric字段其余几个参数的作用和使用方式都与create_llm_as_judge/create_async_llm_as_judge的同名参数相同具体可以参考前面的文章基于OpenEvals的自动化评估-02:LLM-as-a-Judge:让LLM当裁判来评估Agent的输出2. 根据自定义规则进行基于语义的评估接下来我们通过如下几个例子来演示如何利用LLM根据我们自定义的规则对JSON的字段实施评估。我们依然沿用上一篇提供的演示程序为了能够给出评估的理由我们修改了辅助函数pretty_print将EvaluatorResult的comment字段进行了输出。importasynciofromopenevals.jsonimportcreate_async_json_match_evaluatorfromopenevals.typesimportEvaluatorResultfromlangchain_openaiimportChatOpenAIfromdotenvimportload_dotenv load_dotenv()defcreate(name:str,age:int,**kwargs)-dict:person{name:name,age:age}return{**person,**kwargs}defpretty_print(results:EvaluatorResult|list[EvaluatorResult]):if(isinstance(results,list)):forresultinresults:print(f\{result[key]}{result[score]}{result[comment]})else:print(f\{results[key]}{results[score]}{results[comment]})print()asyncdefeval(*,outputs:dict|list[dict],references:dict|list[dict],**kwargs):evaluatorcreate_async_json_match_evaluator(**kwargs)resultsawaitevaluator(outputsoutputs,reference_outputsreferences)pretty_print(results)asyncdefmain():rubric:dict[str,str]{name:conduct strictly string comparison.,aget:conduct strictly numeric comparison.}llmChatOpenAI(modelgpt-5.4-mini)alice1create(Alice,20)alice2create(alice,20)awaiteval(outputsalice1,referencesalice2,rubricrubric,judgellm)如上面的代码所示我们在调用eval函数的时候指定了如下两个参数rubric为评估JSON的两个字段name和age定义了评估规则要求它们进行严格的字符串和数值比较judge指定了一个ChatOpenAI对象用来调用评估模型gpt-5.4-mini如下所示的输出满足我们的预期指定的name由于大小写不一致被判定为不匹配并给出了理由age数值相同视为匹配。json_match:age 1 None json_match:name False The output value is Alice while the expected value is alice. Since the criterion is strict string comparison, the case mismatch means they do not match.现在我们对main函数做了如下的修改在提供的rubric中对字典name的评估规则进行了修改让LLM采用大小写不敏感的方式进行字符串比较。执行之后可以发现两个字段都是匹配的。asyncdefmain():rubric:dict[str,str]{name:conduct string comparison with case-insensitive mode.,aget:conduct strictly numeric comparison.}llmChatOpenAI(modelgpt-5.4-mini)alice1create(Alice,20)alice2create(alice,20)awaiteval(outputsalice1,referencesalice2,rubricrubric,judgellm)输出json_match:age 1 None json_match:name True The output value Alice matches the expected value alice under case-insensitive string comparison.现在我们进一步为两个字典定义如下的评估规则name作为人名可能提供姓Last Name、名First Name和姓名Full Name我们认为只要任意部分匹配就可以了age只要年龄差距不超过5岁也被视为匹配。在如下的演示程序中我将上述的评估规则应用到rubric参数上。在调用eval函数进行评估时传入的两个对象其name和age都不相同但是满足上述的规则所以LLM将它们都判定为匹配asyncdefmain():rubric:dict[str,str]{name:Considered matched if the same first name or last name is mentioned in both output value and referenced value,age:Considered matched if the absolute difference between the output value and referenced value is less than 5.}llmChatOpenAI(modelgpt-5.4-mini)alice1create(Alice,20)alice2create(alice gates,23)awaiteval(outputsalice1,referencesalice2,rubricrubric,judgellm)输出json_match:age True The output age is 20 and the expected age is 23. The absolute difference is 3, which is less than 5, so it matches the criterion. json_match:name True The output value Alice shares the same first name as the expected value alice gates (case-insensitive match), so it satisfies the criterion.3. 忽略指定的字段上一篇和这篇文章基本围绕着create_json_match_evaluator和create_async_json_match_evaluator这两个创建JSON评估器的工厂函数展开介绍我们基本上完整介绍了两个函数的所有参数除了exclude_keys。这个参数的语义很明确就是忽略评估JSON中携带的某些字段。如下面的演示程序所示我在rubric参数中添加一个针对生日字段birth的评估规则并在作为outputs的对象上同时添加了birth和gender两个字段但是利用exclude_keys参数加将后者忽略。所以输出只包含name、age和birth三个字段的评估结果asyncdefmain():rubric:dict[str,str]{name:Considered matched if the same first name or last name is mentioned in both output value and referenced value,age:Considered matched if the absolute difference between the output value and referenced value is less than 5.,birth:Conduct date (without time) based comparison.}llmChatOpenAI(modelgpt-5.4-mini)alice1create(Alice,20,birth1981-08-24,genderfemale)alice2create(alice gates,23,birth24 AUG 1981)awaiteval(outputsalice1,referencesalice2,rubricrubric,judgellm,exclude_keys[gender])输出json_match:age True The output age is 20 and the expected age is 23. The absolute difference is 3, which is less than 5, so it matches the criteria. json_match:birth True The output date 1981-08-24 matches the expected date 24 AUG 1981 when compared by date only, ignoring format and time. json_match:name True The output value Alice shares the same first name as the expected value alice gates (case-insensitive match on Alice/alice), so it matches the criterion.