This site is not available on Mobile. Please return on a desktop browser.
Visit our main site at guardrailsai.com
Developed by | Guardrails AI |
Date of development | Feb 15, 2024 |
Validator type | - |
Blog | |
License | Apache 2 |
Input/Output | Output |
This validator checks that an LLM-generated summary covers the list of topics present in the document.
This validator is only intended for summarization.
Dependencies:
litellm
Foundation model access keys:
$ guardrails hub install hub://guardrails/saliency_check
In this example, we apply the validator to a string output generated by an LLM. Our docs are regarding San Francisco, and the summary is about London. The validator will fail because the summary does not cover the topics present in the document.
# Import Guard and Validator
from guardrails import Guard
from guardrails.hub import SaliencyCheck
# Initialize The Guard with this validator
guard = Guard().use(
SaliencyCheck,
"assets/",
llm_callable="gpt-3.5-turbo",
threshold=0.1,
on_fail="exception",
)
# Test passing response
guard.validate(
"""
San Francisco is a major Californian city, known for its finance, culture, and density.
Originally inhabited by the Yelamu tribe, the city grew rapidly during the Gold Rush and became a major West Coast port.
Despite a devastating earthquake and fire in 1906, San Francisco rebuilt and played significant roles in World War II and international relations.
The city is also known for its liberal activism and social movements.
"""
) # Pass
try:
# Test failing response
guard.validate(
"""
London is a major city and the capital of the United Kingdom. It is located in southeastern England on the River Thames.
London is a leading global city in the arts, commerce, education, entertainment, fashion, finance, healthcare, media, professional services, research and development, tourism, and transportation.
"""
) # Fail
except Exception as e:
print(e)
Output:
Validation failed for field with errors: The summary
Summary:
London is a major city and the capital of the United Kingdom. It is located in southeastern England on the River Thames.
London is a leading global city in the arts, commerce, education, entertainment, fashion, finance, healthcare, media, professional services, research and development, tourism, and transportation.
does not cover these topics:
{'ramaytush ohlone language', 'treaty of san francisco', 'per capita income', 'summer of love', 'united nations charter', 'mission san francisco de asís', 'aggregate income', 'city and county of san francisco', 'land area', 'european settlement', 'san francisco', 'san francisco peninsula', 'california gold rush', 'gay rights movement', 'liberal activism', 'colloquial nicknames', 'northern california', 'yelamu tribe', 'world war ii', '1906 earthquake and fire', 'presidio of san francisco', 'panama-pacific international exposition', 'population'}
__init__(self, docs_dir, llm_callable="gpt-3.5-turbo", threshold=0.25, on_fail="noop")
Initializes a new instance of the Validator class.
Parameters:
docs_dir
(str): Path to the directory containing the documents.llm_callable
(str): The LiteLLM model string name. Default is gpt-3.5-turbo
.threshold
(float): Threshold for overlap between topics in document and summary.on_fail
(str, Callable): The policy to enact when a validator fails. If str
, must be one of reask
, fix
, filter
, refrain
, noop
, exception
or fix_reask
. Otherwise, must be a function that is called when the validator fails.validate(self, value, metadata={}) -> ValidationResult
Validates the given value
using the rules defined in this validator, relying on the metadata
provided to customize the validation process. This method is automatically invoked by guard.parse(...)
, ensuring the validation logic is applied to the input data.
Note:
guard.parse(...)
where this method will be called internally for each associated Validator.guard.parse(...)
, ensure to pass the appropriate metadata
dictionary that includes keys and values required by this validator. If guard
is associated with multiple validators, combine all necessary metadata into a single dictionary.Parameters:
value
(Any): The input value to validate.metadata
(dict): A dictionary containing metadata required for validation. No additional metadata keys are needed for this validator.The validator playground is available to authenticated users. Please log in to use it.