This function will wrap a prompt with a check for a LLM to accept or decline the result of the prompt, providing feedback if the result is declined. The evaluating LLM will be presented with the original prompt and the result of the prompt, and will be asked to verify if the answer is satisfactory (using chain of thought reasoning to arrive at a boolean decision). If the result is declined, the chain of thought responsible for the decision will be summarized and sent back to the original LLM that was asked to evaluate the prompt, so that it may retry the prompt.
Note that this function is experimental and, because it relies on chain of thought reasoning by a LLM about the answer of another LLM, it may not always provide accurate results and can increase the token cost of evaluating a prompt.
Usage
llm_verify(
prompt,
question = "Is the answer satisfactory?",
llm_provider = NULL,
max_words_feedback = 50
)Arguments
- prompt
A single string or a tidyprompt object
- question
The question to ask the LLM to verify the result of the prompt. The LLM will be presented the original prompt, its result, and this question. The LLM will be asked to provide a boolean answer to this question. If TRUE, the result of the prompt will be accepted; if FALSE, the result will be declined
- llm_provider
A llm_provider object which will be used to verify the evaluation led to a satisfactory result. If not provided, the same LLM provider as the prompt was originally evaluated with will be used
- max_words_feedback
The maximum number of words allowed in the summary of why the result was declined. This summary is sent back to the LLM originally asked to evaluate the prompt
Value
A tidyprompt with an added prompt_wrap() which
will add a check for a LLM to accept or decline the result of the prompt,
providing feedback if the result is declined
Details
The original prompt text shown to the LLM is built from the base prompt as well as all prompt wraps that have a modify function but do not have an extraction or validation function. This is to ensure that no redundant validation is performed by the evaluating LLM on instructions which have already been validated by functions in those prompt wraps.
Examples
if (FALSE) { # \dontrun{
"What is 'Enschede'?!" |>
answer_as_text(max_words = 50) |>
llm_verify() |>
send_prompt()
# --- Sending request to LLM provider (gpt-4o-mini): ---
# What is 'Enschede'?!
#
# You must provide a text response. The response must be at most 50 words.
# --- Receiving response from LLM provider: ---
# Enschede is a city in the Netherlands, located in the eastern part near the German border.
# It is known for its vibrant culture, history, and universities, particularly the
# University of Twente, as well as its textiles industry and beautiful parks.
# --- Sending request to LLM provider (gpt-4o-mini): ---
# You are given a user's prompt.
# To answer the user's prompt, you need to think step by step to arrive at a final answer.
#
# ----- START OF USER'S PROMPT -----
# >>> An assistant was asked:
#
# What is 'Enschede'?!
#
# >>> The assistant answered:
#
# [1] "Enschede is a city in the Netherlands, located in the eastern part near
# the German border. It is known for its vibrant culture, history, and universities,
# particularly the University of Twente, as well as its textiles industry and
# beautiful parks."
#
# >>> Is the answer satisfactory?
#
# You must answer with only TRUE or FALSE (use no other characters).
# ----- END OF USER'S PROMPT -----
#
# What are the steps you would take to answer the user's prompt?
# Describe your thought process in the following format:
# >> step 1: <step 1 description>
# >> step 2: <step 2 description>
# (etc.)
#
# When you are done, you must type:
# FINISH[<put here your final answer to the user's prompt>]
#
# Make sure your final answer follows the logical conclusion of your thought process.
# --- Receiving response from LLM provider: ---
# >> step 1: Read the user's prompt carefully to understand what is being asked
# regarding the assistant's answer about Enschede.
# >> step 2: Analyze the assistant's provided answer for accuracy and completeness,
# ensuring it conveys relevant information about Enschede.
# >> step 3: Assess whether the answer includes key aspects, such as the city's
# location, cultural significance, educational institutions, and any notable
# historical references or industries tied to the city.
# >> step 4: Determine if the assistant's answer sufficiently addresses the
# question "What is Enschede?" and provides a clear and informative overview of the city.
# >> step 5: Conclude whether the assistant's answer meets the standards for being
# satisfactory in terms of detail and correctness.
#
# FINISH[TRUE]
# [1] "Enschede is a city in the Netherlands, located in the eastern part near the
# German border. It is known for its vibrant culture, history, and universities,
# particularly the University of Twente, as well as its textiles industry and beautiful parks."
} # }