`langchain.evaluation.parsing.base`.JsonValidityEvaluator¶

class langchain.evaluation.parsing.base.JsonValidityEvaluator(**kwargs: Any)[source]¶

Evaluate whether the prediction is valid JSON.

This evaluator checks if the prediction is a valid JSON string. It does not: require any input or reference.

Parameters: kwargs (Any) –

requires_input¶

Whether this evaluator requires an input string. Always False.

Type: bool

requires_reference¶

Whether this evaluator requires a reference string. Always False.

Type: bool

evaluation_name¶

The name of the evaluation metric. Always “json”.

Type: str

Examples

>>> evaluator = JsonValidityEvaluator()
>>> prediction = '{"name": "John", "age": 30, "city": "New York"}'
>>> evaluator.evaluate(prediction)
{'score': 1}

>>> prediction = '{"name": "John", "age": 30, "city": "New York",}'
>>> evaluator.evaluate(prediction)
{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes'}

Attributes

`evaluation_name`	The name of the evaluation.
`requires_input`	Whether this evaluator requires an input string.
`requires_reference`	Whether this evaluator requires a reference label.

Methods

`__init__`(**kwargs)
`aevaluate_strings`(*, prediction[, ...])	Asynchronously evaluate Chain or LLM output, based on optional input and label.
`evaluate_strings`(*, prediction[, reference, ...])	Evaluate Chain or LLM output, based on optional input and label.

__init__(**kwargs: Any) → None[source]¶

Parameters: kwargs (Any) –
Return type: None

async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶

Asynchronously evaluate Chain or LLM output, based on optional input and label.

Parameters

prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.

Returns

The evaluation results containing the score or value.

Return type

dict

evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶

Evaluate Chain or LLM output, based on optional input and label.