Self Corrective RAG with LangGraph and Groq Llama 3

---

Install dependencies¶

pip install langchain langchain-core langchain-community langchain-groq langgraph wikipedia

Import libraries¶

In [1]:

import warnings
from dotenv import load_dotenv

load_dotenv()
warnings.filterwarnings("ignore")

import warnings from dotenv import load_dotenv load_dotenv() warnings.filterwarnings("ignore")

In [2]:

from typing import Literal, TypedDict
from pydantic import BaseModel, Field
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_groq.chat_models import ChatGroq
from langchain_community.retrievers import WikipediaRetriever
from langgraph.graph import StateGraph, START, END
from IPython.display import Image, display

from typing import Literal, TypedDict from pydantic import BaseModel, Field from langchain_core.documents import Document from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough from langchain_core.output_parsers import StrOutputParser from langchain_groq.chat_models import ChatGroq from langchain_community.retrievers import WikipediaRetriever from langgraph.graph import StateGraph, START, END from IPython.display import Image, display

Basic RAG Components¶

Language Model¶

In [3]:

llm = ChatGroq(
    model='llama3-groq-8b-8192-tool-use-preview',
    temperature=0.45
)

llm_checker = ChatGroq(
    model='llama3-groq-8b-8192-tool-use-preview',
    temperature=0.0
)

res = llm.invoke('Hi! How are you today?')

res.content

llm = ChatGroq( model='llama3-groq-8b-8192-tool-use-preview', temperature=0.45 ) llm_checker = ChatGroq( model='llama3-groq-8b-8192-tool-use-preview', temperature=0.0 ) res = llm.invoke('Hi! How are you today?') res.content

Out[3]:

"I'm doing well, thank you! How can I assist you today?"

Documents Retriever¶

In [4]:

retriever = WikipediaRetriever()

docs = retriever.invoke("Meta AI")

docs[0].metadata

retriever = WikipediaRetriever() docs = retriever.invoke("Meta AI") docs[0].metadata

Out[4]:

{'title': 'Meta AI',
 'summary': "Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies. Meta AI deems itself an academic research laboratory, focused on generating knowledge for the AI community, and should not be confused with Meta's Applied Machine Learning (AML) team, which focuses on the practical applications of its products.\n\n",
 'source': 'https://en.wikipedia.org/wiki/Meta_AI'}

In [5]:

print(docs[0].page_content)

print(docs[0].page_content)

Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies. Meta AI deems itself an academic research laboratory, focused on generating knowledge for the AI community, and should not be confused with Meta's Applied Machine Learning (AML) team, which focuses on the practical applications of its products.


== History ==
The laboratory was founded as Facebook Artificial Intelligence Research (FAIR) with locations at the headquarters in Menlo Park, California, London, United Kingdom, and a new laboratory in Manhattan. FAIR was officially announced in September 2013. FAIR was first directed by New York University's Yann LeCun, a deep learning professor and Turing Award winner. Working with NYU's Center for Data Science, FAIR's initial goal was to research data science, machine learning, and artificial intelligence and to "understand intelligence, to discover its fundamental principles, and to make machines significantly more intelligent". Research at FAIR pioneered the technology that led to face recognition, tagging in photographs, and personalized feed recommendation. Vladimir Vapnik, a pioneer in statistical learning, joined FAIR in 2014. Vapnik is the co-inventor of the support-vector machine and one of the developers of the Vapnik–Chervonenkis theory.
FAIR opened a research center in Paris, France in 2015, and subsequently launched smaller satellite research labs in Seattle, Pittsburgh, Tel Aviv, Montreal and London. In 2016, FAIR partnered with Google, Amazon, IBM, and Microsoft in creating the Partnership on Artificial Intelligence to Benefit People and Society, an organization with a focus on open licensed research, supporting ethical and efficient research practices, and discussing fairness, inclusivity, and transparency.
In 2018, Jérôme Pesenti, former CTO of IBM's big data group, assumed the role of president of FAIR, while LeCun stepped down to serve as chief AI scientist. In 2018, FAIR was placed 25th in the AI Research Rankings 2019, which ranked the top global organizations leading AI research. FAIR quickly rose to eighth position in 2019, and maintained eighth position in the 2020 rank. FAIR had approximately 200 staff in 2018, and had the goal to double that number by 2020.
FAIR's initial work included research in learning-model enabled memory networks, self-supervised learning and generative adversarial networks, text classification and translation, as well as computer vision. FAIR released Torch deep-learning modules as well as PyTorch in 2017, an open-source machine learning framework, which was subsequently used in several deep learning technologies, such as Tesla's autopilot  and Uber's Pyro. Also in 2017, FAIR discontinued a research project once AI bots developed a language that was unintelligible to humans, inciting conversations about dystopian fear of artificial intelligence going out of control. However, FAIR clarified that the research had been shut down because they had accomplished their initial goal to understand how languages are generated, rather than out of fear.
FAIR was renamed Meta AI following the rebranding that changed Facebook, Inc. to Meta Platforms Inc.
In 2022, Meta AI predicted the 3D shape of 600 million potential proteins in two weeks.


== Current research ==


=== Natural language processing and conversational AI ===
Artificial intelligence communication requires a machine to understand natural language and to generate language that is natural. Meta AI seeks to improve these technologies to improve safe communication regardless of what language the user might speak. Thus, a central task involves the generalization of natural language processing (NLP) technology to other languages. As such, Meta AI actively works on unsupervised machine translation. Meta AI seeks to improve natural-language interfaces by developing aspects of chitchat dialogue such as repetition, specificity, response-relat

Tools¶

Format documents¶

In [6]:

def format_docs(docs: list[Document]) -> str:
    formatted = [
        (
            f"Source ID: {i+1}\n"
            f"Article Title: {doc.metadata['title']}\n"
            f"Article URL: {doc.metadata['source']}\n"
            f"Article Content: {doc.page_content}"
        )
        for i, doc in enumerate(docs)
    ]
    return "\n\n" + "\n\n".join(formatted)

def format_docs(docs: list[Document]) -> str: formatted = [ ( f"Source ID: {i+1}\n" f"Article Title: {doc.metadata['title']}\n" f"Article URL: {doc.metadata['source']}\n" f"Article Content: {doc.page_content}" ) for i, doc in enumerate(docs) ] return "\n\n" + "\n\n".join(formatted)

In [7]:

print(format_docs(docs[:2]))

print(format_docs(docs[:2]))

Source ID: 1
Article Title: Meta AI
Article URL: https://en.wikipedia.org/wiki/Meta_AI
Article Content: Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies. Meta AI deems itself an academic research laboratory, focused on generating knowledge for the AI community, and should not be confused with Meta's Applied Machine Learning (AML) team, which focuses on the practical applications of its products.


== History ==
The laboratory was founded as Facebook Artificial Intelligence Research (FAIR) with locations at the headquarters in Menlo Park, California, London, United Kingdom, and a new laboratory in Manhattan. FAIR was officially announced in September 2013. FAIR was first directed by New York University's Yann LeCun, a deep learning professor and Turing Award winner. Working with NYU's Center for Data Science, FAIR's initial goal was to research data science, machine learning, and artificial intelligence and to "understand intelligence, to discover its fundamental principles, and to make machines significantly more intelligent". Research at FAIR pioneered the technology that led to face recognition, tagging in photographs, and personalized feed recommendation. Vladimir Vapnik, a pioneer in statistical learning, joined FAIR in 2014. Vapnik is the co-inventor of the support-vector machine and one of the developers of the Vapnik–Chervonenkis theory.
FAIR opened a research center in Paris, France in 2015, and subsequently launched smaller satellite research labs in Seattle, Pittsburgh, Tel Aviv, Montreal and London. In 2016, FAIR partnered with Google, Amazon, IBM, and Microsoft in creating the Partnership on Artificial Intelligence to Benefit People and Society, an organization with a focus on open licensed research, supporting ethical and efficient research practices, and discussing fairness, inclusivity, and transparency.
In 2018, Jérôme Pesenti, former CTO of IBM's big data group, assumed the role of president of FAIR, while LeCun stepped down to serve as chief AI scientist. In 2018, FAIR was placed 25th in the AI Research Rankings 2019, which ranked the top global organizations leading AI research. FAIR quickly rose to eighth position in 2019, and maintained eighth position in the 2020 rank. FAIR had approximately 200 staff in 2018, and had the goal to double that number by 2020.
FAIR's initial work included research in learning-model enabled memory networks, self-supervised learning and generative adversarial networks, text classification and translation, as well as computer vision. FAIR released Torch deep-learning modules as well as PyTorch in 2017, an open-source machine learning framework, which was subsequently used in several deep learning technologies, such as Tesla's autopilot  and Uber's Pyro. Also in 2017, FAIR discontinued a research project once AI bots developed a language that was unintelligible to humans, inciting conversations about dystopian fear of artificial intelligence going out of control. However, FAIR clarified that the research had been shut down because they had accomplished their initial goal to understand how languages are generated, rather than out of fear.
FAIR was renamed Meta AI following the rebranding that changed Facebook, Inc. to Meta Platforms Inc.
In 2022, Meta AI predicted the 3D shape of 600 million potential proteins in two weeks.


== Current research ==


=== Natural language processing and conversational AI ===
Artificial intelligence communication requires a machine to understand natural language and to generate language that is natural. Meta AI seeks to improve these technologies to improve safe communication regardless of what language the user might speak. Thus, a central task involves the generalization of natural language processing (NLP) technology to other languages. As such, Meta AI actively works on unsupervised machine translation. Meta AI seeks to improve natural-language interfaces by developing aspects of chitchat dialogue such as repetition, specificity, response-relat

Source ID: 2
Article Title: Llama (language model)
Article URL: https://en.wikipedia.org/wiki/Llama_(language_model)
Article Content: Llama (acronym for Large Language Model Meta AI, and formerly stylized as LLaMA) is a family of autoregressive large language models (LLMs) released by Meta AI starting in February 2023. The latest version is Llama 3.1, released in July 2024.
Model weights for the first version of Llama were made available to the research community under a non-commercial license, and access was granted on a case-by-case basis. Unauthorized copies of the model were shared via BitTorrent. In response, Meta AI issued DMCA takedown requests against repositories sharing the link on GitHub. Subsequent versions of Llama were made accessible outside academia and released under licenses that permitted some commercial use. Llama models are trained at different parameter sizes, ranging between 7B and 405B. Originally, Llama was only available as a foundation model. Starting with Llama 2, Meta AI started releasing instruction fine-tuned versions alongside foundation models.
Alongside the release of Llama 3, Meta added virtual assistant features to Facebook and WhatsApp in select regions, and a standalone website. Both services use a Llama 3 model.


== Background ==
After the release of large language models such as GPT-3, a focus of research was up-scaling models which in some instances showed major increases in emergent capabilities. The release of ChatGPT and its surprise success caused an increase in attention to large language models.
Compared with other responses to ChatGPT, Meta's Chief AI scientist Yann LeCun stated that large language models are best for aiding with writing.
An empirical investigation of the Llama series was the scaling laws. It was observed that the Llama 3 models showed that when a model is trained on data that is more than the "Chinchilla-optimal" amount, the performance continues to scale log-linearly. For example, the Chinchilla-optimal dataset for Llama 3 8B is 200 billion tokens, but performance continued to scale log-linearly to the 75-times larger dataset of 15 trillion tokens.


== Initial release ==
LLaMA was announced on February 24, 2023, via a blog post and a paper describing the model's training, architecture, and performance. The inference code used to run the model was publicly released under the open-source GPLv3 license. Access to the model's weights was managed by an application process, with access to be granted "on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world".
Llama was trained on only publicly available information, and was trained at various model sizes, with the intention to make it more accessible to different hardware.
Meta AI reported the 13B parameter model performance on most NLP benchmarks exceeded that of the much larger GPT-3 (with 175B parameters), and the largest 65B model was competitive with state of the art models such as PaLM and Chinchilla.


=== Leak ===
On March 3, 2023, a torrent containing LLaMA's weights was uploaded, with a link to the torrent shared on the 4chan imageboard and subsequently spread through online AI communities. That same day, a pull request on the main LLaMA repository was opened, requesting to add the magnet link to the official documentation. On March 4, a pull request was opened to add links to HuggingFace repositories containing the model. On March 6, Meta filed takedown requests to remove the HuggingFace repositories linked in the pull request, characterizing it as "unauthorized distribution" of the model. HuggingFace complied with the requests. On March 20, Meta filed a DMCA takedown request for copyright infringement against a repository containing a script that downloaded LLaMA from a mirror, and GitHub complied the next day.
Reactions to the leak varied. Some speculated that the model would be used for malicious purposes, such as more sophisticated spam. Some have celebrated the model's accessibility, as well as the fact that s

Chains¶

Format documents chain¶

In [8]:

format_docs_chain = RunnablePassthrough.assign(
    documents=(lambda x: format_docs(x["documents"]))
)

format_docs_chain = RunnablePassthrough.assign( documents=(lambda x: format_docs(x["documents"])) )

Answer question with citations¶

CitedAnswer Output Model¶

In [9]:

class CitedAnswer(BaseModel):
    """Answer the user question based only on the given sources, and cite the sources used."""

    answer: str = Field(
        ...,
        description="The answer to the user question, which is based only on the given sources.",
    )
    citations: list[int] = Field(
        ...,
        description="The integer IDs of the SPECIFIC sources which justify the answer.",
    )

class CitedAnswer(BaseModel): """Answer the user question based only on the given sources, and cite the sources used.""" answer: str = Field( ..., description="The answer to the user question, which is based only on the given sources.", ) citations: list[int] = Field( ..., description="The integer IDs of the SPECIFIC sources which justify the answer.", )

Prompt¶

In [10]:

RAG_SYSTEM_PROMPT = (
    "You are a helpful AI assistant. Your task is to answer questions using ONLY the information provided in the given articles.\n\n"
    "Instructions:\n"
    "1. Answer the user's question using ONLY the information from the provided articles.\n"
    "2. Cite the source for EVERY statement using the format [source_id] at the end of each sentence.\n"
    "3. If the provided sources do not contain enough information to answer the question, respond: 'I don't have enough information in the provided sources to answer this question.'\n"
    "4. DO NOT use external knowledge or any information not included in the provided articles, even if it seems correct.\n"
    "5. If you can only partially answer the question, provide the available information and indicate that the rest cannot be answered with the provided sources.\n"
    "6. Be concise and direct in your responses.\n\n"
    "Reference articles:\n"
    "{documents}\n\n"
)


RAG_PROMPT = ChatPromptTemplate.from_messages(
    [
        ("system", RAG_SYSTEM_PROMPT),
        ("human", "{question}"),
    ]
)

RAG_PROMPT.pretty_print()

RAG_SYSTEM_PROMPT = ( "You are a helpful AI assistant. Your task is to answer questions using ONLY the information provided in the given articles.\n\n" "Instructions:\n" "1. Answer the user's question using ONLY the information from the provided articles.\n" "2. Cite the source for EVERY statement using the format [source_id] at the end of each sentence.\n" "3. If the provided sources do not contain enough information to answer the question, respond: 'I don't have enough information in the provided sources to answer this question.'\n" "4. DO NOT use external knowledge or any information not included in the provided articles, even if it seems correct.\n" "5. If you can only partially answer the question, provide the available information and indicate that the rest cannot be answered with the provided sources.\n" "6. Be concise and direct in your responses.\n\n" "Reference articles:\n" "{documents}\n\n" ) RAG_PROMPT = ChatPromptTemplate.from_messages( [ ("system", RAG_SYSTEM_PROMPT), ("human", "{question}"), ] ) RAG_PROMPT.pretty_print()

================================ System Message ================================

You are a helpful AI assistant. Your task is to answer questions using ONLY the information provided in the given articles.

Instructions:
1. Answer the user's question using ONLY the information from the provided articles.
2. Cite the source for EVERY statement using the format [source_id] at the end of each sentence.
3. If the provided sources do not contain enough information to answer the question, respond: 'I don't have enough information in the provided sources to answer this question.'
4. DO NOT use external knowledge or any information not included in the provided articles, even if it seems correct.
5. If you can only partially answer the question, provide the available information and indicate that the rest cannot be answered with the provided sources.
6. Be concise and direct in your responses.

Reference articles:
{documents}



================================ Human Message =================================

{question}

RAG Chain¶

In [11]:

rag_llm = llm.with_structured_output(CitedAnswer)

rag_response_chain = (
    format_docs_chain
    | RAG_PROMPT
    | rag_llm
)

retrieve_docs = (lambda x: x['question']) | retriever

rag_chain = RunnablePassthrough.assign(
    documents=retrieve_docs
).assign(response=rag_response_chain)

rag_llm = llm.with_structured_output(CitedAnswer) rag_response_chain = ( format_docs_chain | RAG_PROMPT | rag_llm ) retrieve_docs = (lambda x: x['question']) | retriever rag_chain = RunnablePassthrough.assign( documents=retrieve_docs ).assign(response=rag_response_chain)

In [12]:

result = rag_chain.invoke({"question": "What is Meta AI?"})

result['response']

result = rag_chain.invoke({"question": "What is Meta AI?"}) result['response']

Out[12]:

CitedAnswer(answer='Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies.', citations=[1])

In [13]:

result = rag_chain.invoke({"question": "Who is Jann LeCun?"})

result['response']

result = rag_chain.invoke({"question": "Who is Jann LeCun?"}) result['response']

Out[13]:

CitedAnswer(answer="I don't have enough information in the provided sources to answer this question.", citations=[])

Rewrite queries¶

Prompt¶

In [14]:

QUERY_REWRITER_SYSTEM = (
    "You are a query rewriter that enhances input queries for optimal Wikipedia search and retrieval. "
    "Your task is to generate ONE new query that is semantically related to the original and remains within the same domain. "
    "Do not provide explanations, ask questions, or include any additional text other than the new query. "
    "Analyze the input to understand its underlying intent and then output only the improved query."
)

QUERY_REWRITER_PROMPT = ChatPromptTemplate.from_messages(
    [
        ("system", QUERY_REWRITER_SYSTEM),
        (
            "human",
            "{query}",
        ),
    ]
)

QUERY_REWRITER_PROMPT.pretty_print()

QUERY_REWRITER_SYSTEM = ( "You are a query rewriter that enhances input queries for optimal Wikipedia search and retrieval. " "Your task is to generate ONE new query that is semantically related to the original and remains within the same domain. " "Do not provide explanations, ask questions, or include any additional text other than the new query. " "Analyze the input to understand its underlying intent and then output only the improved query." ) QUERY_REWRITER_PROMPT = ChatPromptTemplate.from_messages( [ ("system", QUERY_REWRITER_SYSTEM), ( "human", "{query}", ), ] ) QUERY_REWRITER_PROMPT.pretty_print()

================================ System Message ================================

You are a query rewriter that enhances input queries for optimal Wikipedia search and retrieval. Your task is to generate ONE new query that is semantically related to the original and remains within the same domain. Do not provide explanations, ask questions, or include any additional text other than the new query. Analyze the input to understand its underlying intent and then output only the improved query.

================================ Human Message =================================

{query}

Query Rewriter Chain¶

In [15]:

query_rewriter_chain = QUERY_REWRITER_PROMPT | llm | StrOutputParser()

result = query_rewriter_chain.invoke({"query": "What is Meta AI?"})

result

query_rewriter_chain = QUERY_REWRITER_PROMPT | llm | StrOutputParser() result = query_rewriter_chain.invoke({"query": "What is Meta AI?"}) result

Out[15]:

'Meta AI research labs'

Check Context Relevance¶

CheckContextResponse Output Model¶

In [16]:

class CheckContextResponse(BaseModel):
    """Decide if the documents are relevant to the query."""

    binary_score: Literal["yes", "no"] = Field(
        description="Documents are relevants, 'yes' or 'no'"
    )

class CheckContextResponse(BaseModel): """Decide if the documents are relevant to the query.""" binary_score: Literal["yes", "no"] = Field( description="Documents are relevants, 'yes' or 'no'" )

Prompt¶

In [17]:

CHECK_CONTEXT_SYSTEM = (
    "You are a relevance evaluator for documents. "
    "Given a set of documents and a query, your task is to determine the relevance of each document to the query. "
    "Respond with yes if a document is relevant to the query and no if it is not. "
)

CHECK_CONTEXT_PROMPT = ChatPromptTemplate.from_messages(
    [
        ('system', CHECK_CONTEXT_SYSTEM),
        ('user', "DOCUMENTS:\n\n{documents}\n\nQUERY: {query}\n\n"),
    ]
)

CHECK_CONTEXT_PROMPT.pretty_print()

CHECK_CONTEXT_SYSTEM = ( "You are a relevance evaluator for documents. " "Given a set of documents and a query, your task is to determine the relevance of each document to the query. " "Respond with yes if a document is relevant to the query and no if it is not. " ) CHECK_CONTEXT_PROMPT = ChatPromptTemplate.from_messages( [ ('system', CHECK_CONTEXT_SYSTEM), ('user', "DOCUMENTS:\n\n{documents}\n\nQUERY: {query}\n\n"), ] ) CHECK_CONTEXT_PROMPT.pretty_print()

================================ System Message ================================

You are a relevance evaluator for documents. Given a set of documents and a query, your task is to determine the relevance of each document to the query. Respond with yes if a document is relevant to the query and no if it is not. 

================================ Human Message =================================

DOCUMENTS:

{documents}

QUERY: {query}

Check Context Relevance Chain¶

In [18]:

check_context_llm = llm_checker.with_structured_output(CheckContextResponse)
check_context_chain = (
    format_docs_chain
    | CHECK_CONTEXT_PROMPT
    | check_context_llm
)

check_context_llm = llm_checker.with_structured_output(CheckContextResponse) check_context_chain = ( format_docs_chain | CHECK_CONTEXT_PROMPT | check_context_llm )

In [19]:

docs = retriever.invoke("Meta AI")

result = check_context_chain.invoke(
    {"documents": docs, "query": "What is Meta AI?"}
)

result

docs = retriever.invoke("Meta AI") result = check_context_chain.invoke( {"documents": docs, "query": "What is Meta AI?"} ) result

Out[19]:

CheckContextResponse(binary_score='yes')

In [20]:

result = check_context_chain.invoke(
    {"documents": docs, "query": "Who is Miyamoto Musashi?"}
)

result

result = check_context_chain.invoke( {"documents": docs, "query": "Who is Miyamoto Musashi?"} ) result

Out[20]:

CheckContextResponse(binary_score='no')

Check Hallucinations¶

CheckHallucinations Output Model¶

In [21]:

class CheckHallucinations(BaseModel):
    """Binary score for hallucination present in generation answer."""

    binary_score: Literal["yes", "no"] = Field(
        description="Answer is grounded in the facts, 'yes' or 'no'"
    )

class CheckHallucinations(BaseModel): """Binary score for hallucination present in generation answer.""" binary_score: Literal["yes", "no"] = Field( description="Answer is grounded in the facts, 'yes' or 'no'" )

Prompt¶

In [22]:

CHECK_HALLUCINATION_SYSTEM = (
    "You are a grader assessing whether an LLM's response is grounded in and supported by a set of retrieved facts. "
    "Provide a binary score of 'yes' or 'no'. A 'yes' score means the response is fully grounded in and supported by the provided facts, while a 'no' score means it is not. "
    "If the response includes code examples, ensure those examples are fully present in the set of facts; otherwise, always return a score of 'no'."
)

CHECK_HALLUCINATION_PROMPT = ChatPromptTemplate.from_messages(
    [
        ("system", CHECK_HALLUCINATION_SYSTEM),
        ("human",
         "Set of facts: \n\n {documents} \n\n LLM generation: {generation}"),
    ]
)

CHECK_HALLUCINATION_PROMPT.pretty_print()

CHECK_HALLUCINATION_SYSTEM = ( "You are a grader assessing whether an LLM's response is grounded in and supported by a set of retrieved facts. " "Provide a binary score of 'yes' or 'no'. A 'yes' score means the response is fully grounded in and supported by the provided facts, while a 'no' score means it is not. " "If the response includes code examples, ensure those examples are fully present in the set of facts; otherwise, always return a score of 'no'." ) CHECK_HALLUCINATION_PROMPT = ChatPromptTemplate.from_messages( [ ("system", CHECK_HALLUCINATION_SYSTEM), ("human", "Set of facts: \n\n {documents} \n\n LLM generation: {generation}"), ] ) CHECK_HALLUCINATION_PROMPT.pretty_print()

================================ System Message ================================

You are a grader assessing whether an LLM's response is grounded in and supported by a set of retrieved facts. Provide a binary score of 'yes' or 'no'. A 'yes' score means the response is fully grounded in and supported by the provided facts, while a 'no' score means it is not. If the response includes code examples, ensure those examples are fully present in the set of facts; otherwise, always return a score of 'no'.

================================ Human Message =================================

Set of facts: 

 {documents} 

 LLM generation: {generation}

Check Hallucinations Chain¶

In [23]:

check_hallucination_llm = llm_checker.with_structured_output(
    CheckHallucinations)

check_hallucination_chain = (
    format_docs_chain
    | CHECK_HALLUCINATION_PROMPT
    | check_hallucination_llm
)

check_hallucination_llm = llm_checker.with_structured_output( CheckHallucinations) check_hallucination_chain = ( format_docs_chain | CHECK_HALLUCINATION_PROMPT | check_hallucination_llm )

In [24]:

generation = 'Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies.'

result = check_hallucination_chain.invoke(
    {"documents": docs, "generation": generation}
)

result

generation = 'Meta AI is an American company owned by Meta (formerly Facebook) that develops artificial intelligence and augmented and artificial reality technologies.' result = check_hallucination_chain.invoke( {"documents": docs, "generation": generation} ) result

Out[24]:

CheckHallucinations(binary_score='yes')

In [25]:

generation = 'Miyamoto Musashi was legendary Japanese hero, artist and swordsman.'

result = check_hallucination_chain.invoke(
    {"documents": docs, "generation": generation}
)

result

generation = 'Miyamoto Musashi was legendary Japanese hero, artist and swordsman.' result = check_hallucination_chain.invoke( {"documents": docs, "generation": generation} ) result

Out[25]:

CheckHallucinations(binary_score='yes')

Check Answer Quality¶

CheckAnswer Output Model¶

In [26]:

class CheckAnswer(BaseModel):
    """Binary score to assess answer addresses question."""
    binary_score: Literal["yes", "no"] = Field(
        description="Answer addresses the question, 'yes' or 'no'"
    )

class CheckAnswer(BaseModel): """Binary score to assess answer addresses question.""" binary_score: Literal["yes", "no"] = Field( description="Answer addresses the question, 'yes' or 'no'" )

Prompt¶

In [27]:

CHECK_ANSWER_SYSTEM = (
    "You are a grader assessing whether an answer adequately addresses and resolves a given query. "
    "Provide a binary score of 'yes' or 'no'. A 'yes' score indicates that the answer fully resolves the query, while a 'no' score indicates that it does not."
)

CHECK_ANSWER_PROMPT = ChatPromptTemplate.from_messages(
    [
        ("system", CHECK_ANSWER_SYSTEM),
        ("human",
         "User query: {query}\n\nLLM generation: {generation}"),
    ]
)

CHECK_ANSWER_PROMPT.pretty_print()

CHECK_ANSWER_SYSTEM = ( "You are a grader assessing whether an answer adequately addresses and resolves a given query. " "Provide a binary score of 'yes' or 'no'. A 'yes' score indicates that the answer fully resolves the query, while a 'no' score indicates that it does not." ) CHECK_ANSWER_PROMPT = ChatPromptTemplate.from_messages( [ ("system", CHECK_ANSWER_SYSTEM), ("human", "User query: {query}\n\nLLM generation: {generation}"), ] ) CHECK_ANSWER_PROMPT.pretty_print()

================================ System Message ================================

You are a grader assessing whether an answer adequately addresses and resolves a given query. Provide a binary score of 'yes' or 'no'. A 'yes' score indicates that the answer fully resolves the query, while a 'no' score indicates that it does not.

================================ Human Message =================================

User query: {query}

LLM generation: {generation}

Check Answer Chain¶

In [28]:

check_answer_llm = llm_checker.with_structured_output(CheckAnswer)

check_answer_chain = CHECK_ANSWER_PROMPT | check_answer_llm

check_answer_llm = llm_checker.with_structured_output(CheckAnswer) check_answer_chain = CHECK_ANSWER_PROMPT | check_answer_llm

In [29]:

generation = 'Jann LeCun is a French computer scientist and director of AI Research at Facebook. He is known for his work on convolutional neural networks and is a co-recipient of the Turing Award.'

result = check_answer_chain.invoke(
    {"query": "What is Jann LeCun?", "generation": generation}
)

result

generation = 'Jann LeCun is a French computer scientist and director of AI Research at Facebook. He is known for his work on convolutional neural networks and is a co-recipient of the Turing Award.' result = check_answer_chain.invoke( {"query": "What is Jann LeCun?", "generation": generation} ) result

Out[29]:

CheckAnswer(binary_score='yes')

In [30]:

result = check_answer_chain.invoke(
    {"query": "Who is Miyamoto Musashi?", "generation": generation}
)

result

result = check_answer_chain.invoke( {"query": "Who is Miyamoto Musashi?", "generation": generation} ) result

Out[30]:

CheckAnswer(binary_score='no')

LangGraph Workflow¶

Graph State¶

In [31]:

class GraphState(TypedDict):
    query: str
    requery: str
    can_requery: int
    can_retry: int
    documents: list[Document]
    response: CitedAnswer
    is_fail: bool

class GraphState(TypedDict): query: str requery: str can_requery: int can_retry: int documents: list[Document] response: CitedAnswer is_fail: bool

Graph Nodes¶

Debug tools¶

In [32]:

VERBOSE = True


def print_node(node_name: str):
    if VERBOSE:
        print("==============================")
        print(node_name)
        print("==============================")


def print_result(data):
    if VERBOSE:
        print("------------------------------")
        print(data)
        print("------------------------------")

VERBOSE = True def print_node(node_name: str): if VERBOSE: print("==============================") print(node_name) print("==============================") def print_result(data): if VERBOSE: print("------------------------------") print(data) print("------------------------------")

Documents retriever¶

In [33]:

def documents_retriever(state: GraphState) -> GraphState:
    print_node(documents_retriever.__name__)
    query = state['requery'] if state.get('requery') else state['query']
    documents = retriever.invoke(query)

    return {"documents": documents}

def documents_retriever(state: GraphState) -> GraphState: print_node(documents_retriever.__name__) query = state['requery'] if state.get('requery') else state['query'] documents = retriever.invoke(query) return {"documents": documents}

Query Rewriter¶

In [34]:

def query_rewriter(state: GraphState) -> GraphState:
    print_node(query_rewriter.__name__)
    query = state['query']
    requery = query_rewriter_chain.invoke({"query": query})
    print_result(requery)

    return {"requery": requery, "can_requery": max(0, state['can_requery'] - 1)}

def query_rewriter(state: GraphState) -> GraphState: print_node(query_rewriter.__name__) query = state['query'] requery = query_rewriter_chain.invoke({"query": query}) print_result(requery) return {"requery": requery, "can_requery": max(0, state['can_requery'] - 1)}

Query Answerer¶

In [35]:

def query_answerer(state: GraphState) -> GraphState:
    print_node(query_answerer.__name__)
    query = state['query']
    documents = state['documents']

    response = rag_response_chain.invoke(
        {"question": query, "documents": documents}
    )

    print_result(response)

    return {"response": response}


def fail_answer(state: GraphState) -> GraphState:
    return {"is_fail": True}

def query_answerer(state: GraphState) -> GraphState: print_node(query_answerer.__name__) query = state['query'] documents = state['documents'] response = rag_response_chain.invoke( {"question": query, "documents": documents} ) print_result(response) return {"response": response} def fail_answer(state: GraphState) -> GraphState: return {"is_fail": True}

Retry¶

In [36]:

def retry(state: GraphState) -> GraphState:
    print_node(retry.__name__)
    return {"can_retry": max(0, state['can_retry'] - 1)}

def retry(state: GraphState) -> GraphState: print_node(retry.__name__) return {"can_retry": max(0, state['can_retry'] - 1)}

Graph Conditional Edges¶

Context Checker¶

In [37]:

def context_checker(state: GraphState) -> Literal['query_rewriter', 'query_answerer', 'fail_answer']:
    print_node(context_checker.__name__)

    if len(state['documents']) == 0:
        print_result('No relevant documents found. Failing answer.')
        return 'fail_answer'

    if state['can_requery']:
        query = state['query']
        documents = state['documents']

        result = check_context_chain.invoke(
            {"query": query, "documents": documents}
        )

        print_result(result)

        if result.binary_score == 'no':
            return 'query_rewriter'
        else:
            return 'query_answerer'

    return 'query_answerer'

def context_checker(state: GraphState) -> Literal['query_rewriter', 'query_answerer', 'fail_answer']: print_node(context_checker.__name__) if len(state['documents']) == 0: print_result('No relevant documents found. Failing answer.') return 'fail_answer' if state['can_requery']: query = state['query'] documents = state['documents'] result = check_context_chain.invoke( {"query": query, "documents": documents} ) print_result(result) if result.binary_score == 'no': return 'query_rewriter' else: return 'query_answerer' return 'query_answerer'

Answer Checker¶

In [38]:

def answer_checker(state: GraphState) -> Literal['fail_answer', 'retry', '__end__']:
    print_node(answer_checker.__name__)
    query = state['query']
    documents = state['documents']
    response = state['response']
    generation = response.answer

    result = check_hallucination_chain.invoke(
        {"documents": documents, "generation": generation}
    )

    print_result(f'Pass Hallucination Check: {result.binary_score}')

    if result.binary_score == 'no':
        if state['can_retry']:
            return 'retry'
        else:
            return 'fail_answer'

    result = check_answer_chain.invoke(
        {"query": query, "generation": generation}
    )

    print_result(f'Pass Answer Check: {result.binary_score}')

    if result.binary_score == 'no':
        if state['can_retry']:
            return 'retry'
        else:
            return 'fail_answer'

    return '__end__'

def answer_checker(state: GraphState) -> Literal['fail_answer', 'retry', '__end__']: print_node(answer_checker.__name__) query = state['query'] documents = state['documents'] response = state['response'] generation = response.answer result = check_hallucination_chain.invoke( {"documents": documents, "generation": generation} ) print_result(f'Pass Hallucination Check: {result.binary_score}') if result.binary_score == 'no': if state['can_retry']: return 'retry' else: return 'fail_answer' result = check_answer_chain.invoke( {"query": query, "generation": generation} ) print_result(f'Pass Answer Check: {result.binary_score}') if result.binary_score == 'no': if state['can_retry']: return 'retry' else: return 'fail_answer' return '__end__'

Assemble Graph¶

In [39]:

workflow = StateGraph(GraphState)

# Nodes
workflow.add_node(documents_retriever.__name__, documents_retriever)
workflow.add_node(query_rewriter.__name__, query_rewriter)
workflow.add_node(query_answerer.__name__, query_answerer)
workflow.add_node(retry.__name__, retry)
workflow.add_node(fail_answer.__name__, fail_answer)

# Edges
workflow.add_edge(START, documents_retriever.__name__)
workflow.add_conditional_edges(documents_retriever.__name__, context_checker)
workflow.add_edge(query_rewriter.__name__, documents_retriever.__name__)
workflow.add_conditional_edges(query_answerer.__name__, answer_checker)
workflow.add_edge(retry.__name__, query_rewriter.__name__)
workflow.add_edge(fail_answer.__name__, END)

graph = workflow.compile()

display(Image(graph.get_graph().draw_mermaid_png()))

workflow = StateGraph(GraphState) # Nodes workflow.add_node(documents_retriever.__name__, documents_retriever) workflow.add_node(query_rewriter.__name__, query_rewriter) workflow.add_node(query_answerer.__name__, query_answerer) workflow.add_node(retry.__name__, retry) workflow.add_node(fail_answer.__name__, fail_answer) # Edges workflow.add_edge(START, documents_retriever.__name__) workflow.add_conditional_edges(documents_retriever.__name__, context_checker) workflow.add_edge(query_rewriter.__name__, documents_retriever.__name__) workflow.add_conditional_edges(query_answerer.__name__, answer_checker) workflow.add_edge(retry.__name__, query_rewriter.__name__) workflow.add_edge(fail_answer.__name__, END) graph = workflow.compile() display(Image(graph.get_graph().draw_mermaid_png()))

Run¶

In [40]:

async def run(query: str, verbose=True) -> GraphState:
    initial_state: GraphState = {
        "query": query,
        "can_requery": 1,
        "can_retry": 1,
    }

    return await graph.ainvoke(initial_state)

async def run(query: str, verbose=True) -> GraphState: initial_state: GraphState = { "query": query, "can_requery": 1, "can_retry": 1, } return await graph.ainvoke(initial_state)

In [41]:

result = await run("Who was Miyamoto Musashi?")

result['response']

result = await run("Who was Miyamoto Musashi?") result['response']

==============================
documents_retriever
==============================
==============================
context_checker
==============================
------------------------------
binary_score='yes'
------------------------------
==============================
query_answerer
==============================
------------------------------
answer='Miyamoto Musashi was a Japanese swordsman, strategist, artist, and writer who became renowned through stories of his unique double-bladed swordsmanship and undefeated record in his 62 duels.' citations=[3]
------------------------------
==============================
answer_checker
==============================
------------------------------
Pass Hallucination Check: yes
------------------------------
------------------------------
Pass Answer Check: yes
------------------------------

Out[41]:

CitedAnswer(answer='Miyamoto Musashi was a Japanese swordsman, strategist, artist, and writer who became renowned through stories of his unique double-bladed swordsmanship and undefeated record in his 62 duels.', citations=[3])

In [42]:

result = await run("What is the answer to the Great Question of Life, the Universe, and Everything?")

result = await run("What is the answer to the Great Question of Life, the Universe, and Everything?")

==============================
documents_retriever
==============================
==============================
context_checker
==============================
------------------------------
binary_score='yes'
------------------------------
==============================
query_answerer
==============================
------------------------------
answer='42' citations=[1]
------------------------------
==============================
answer_checker
==============================
------------------------------
Pass Hallucination Check: yes
------------------------------
------------------------------
Pass Answer Check: yes
------------------------------

In [43]:

result['response']

result['response']

Out[43]:

CitedAnswer(answer='42', citations=[1])

In [44]:

result

result

Out[44]:

{'query': 'What is the answer to the Great Question of Life, the Universe, and Everything?',
 'can_requery': 1,
 'can_retry': 1,
 'documents': [Document(metadata={'title': 'Meaning of life', 'summary': 'The meaning of life pertains to the inherent significance or philosophical meaning of living (or existence in general). There is no consensus on a definitive answer, and thinking or discourse on the topic is sought in the English language through the question, "What is the meaning of life?" (or the related "Why are we here?" or "What is the purpose of existence?"). There have been many proposed answers to these questions from many different cultural and ideological backgrounds. The search for life\'s meaning has produced much philosophical, scientific, theological, and metaphysical speculation throughout history. Different people and cultures believe different things for the answer to this question. Opinions vary on the usefulness of using time and resources in the pursuit of an answer. Excessive pondering can be indicative of, or lead to, an existential crisis.\nThe meaning of life can be derived from philosophical and religious contemplation of, and scientific inquiries about, existence, social ties, consciousness, and happiness. Many other issues are also involved, such as symbolic meaning, ontology, value, purpose, ethics, good and evil, free will, the existence of one or multiple gods, conceptions of God, the soul, and the afterlife. Scientific contributions focus primarily on describing related empirical facts about the universe, exploring the context and parameters concerning the "how" of life. Science also studies and can provide recommendations for the pursuit of well-being and a related conception of morality. An alternative, humanistic approach poses the question, "What is the meaning of my life?"\n\n', 'source': 'https://en.wikipedia.org/wiki/Meaning_of_life'}, page_content='The meaning of life pertains to the inherent significance or philosophical meaning of living (or existence in general). There is no consensus on a definitive answer, and thinking or discourse on the topic is sought in the English language through the question, "What is the meaning of life?" (or the related "Why are we here?" or "What is the purpose of existence?"). There have been many proposed answers to these questions from many different cultural and ideological backgrounds. The search for life\'s meaning has produced much philosophical, scientific, theological, and metaphysical speculation throughout history. Different people and cultures believe different things for the answer to this question. Opinions vary on the usefulness of using time and resources in the pursuit of an answer. Excessive pondering can be indicative of, or lead to, an existential crisis.\nThe meaning of life can be derived from philosophical and religious contemplation of, and scientific inquiries about, existence, social ties, consciousness, and happiness. Many other issues are also involved, such as symbolic meaning, ontology, value, purpose, ethics, good and evil, free will, the existence of one or multiple gods, conceptions of God, the soul, and the afterlife. Scientific contributions focus primarily on describing related empirical facts about the universe, exploring the context and parameters concerning the "how" of life. Science also studies and can provide recommendations for the pursuit of well-being and a related conception of morality. An alternative, humanistic approach poses the question, "What is the meaning of my life?"\n\n\n== Origin of the expression ==\n\nThe first English use of the expression "meaning of life" appears in Thomas Carlyle\'s Sartor Resartus (1833–1834), book II chapter IX, "The Everlasting Yea". Our Life is compassed round with Necessity; yet is the meaning of Life itself no other than Freedom, than Voluntary Force: thus have we a warfare; in the beginning, especially, a hard-fought battle.Carlyle may have been inspired by earlier usage of the equivalent German expression der Sinn des Lebens by German Romantic writers Novalis and Friedrich Schlegel. Schlegel was the first to use it in print by way of his novel Lucinde (1799), though Novalis had done so in a 1797–1798 manuscript, in which he wrote: "Only an artist can divine the meaning of life." Additionally, the word lebenssinn, translated as life\'s meaning, had been used by Goethe in a 1796 letter to Schiller. These authors grappled with the rationalism and materialism of modernity. Carlyle called this the "Torch of Science", which burned "more fiercely than ever" and made religion "all parched away, under the Droughts of practical and spiritual Unbelief", resulting in the "Wilderness" of "the wide World in an Atheistic Century".\n\n\n== Origin of the question ==\nArthur Schopenhauer was the first to explicitly ask the question, in an essay entitled "Character".Since a man does not alter, and his moral character remains absolutely the same all through his life; since he must play out the part which he has received, without the least deviation from the character; since neither experience, nor philosophy, nor religion can effect any improvement in him, the question arises, What is the meaning of life at all? To what purpose is it played, this farce in which everything that is essential is irrevocably fixed and determined?Questions about the meaning of life, and similar, have been expressed in a broad variety of other ways, including:\nWhat is the meaning of life? What\'s it all about? Who are we?\nWhy are we here? What are we here for?\nWhat is the origin of life?\nWhat is the nature of life? What is the nature of reality?\nWhat is the purpose of life? What is the purpose of one\'s life?\nWhat is the significance of life? (See also #Psychological significance and value in life)\nWhat is meaningful and valuable in life?\nWhat is the value of life?\nWhat is the reason to live? What are we livin'),
  Document(metadata={'title': "List of The Hitchhiker's Guide to the Galaxy characters", 'summary': "This page is a list of characters in The Hitchhiker's Guide to the Galaxy, by Douglas Adams. The descriptions of the characters are accompanied by information on details about appearances and references to the characters.", 'source': 'https://en.wikipedia.org/wiki/List_of_The_Hitchhiker%27s_Guide_to_the_Galaxy_characters'}, page_content='This page is a list of characters in The Hitchhiker\'s Guide to the Galaxy, by Douglas Adams. The descriptions of the characters are accompanied by information on details about appearances and references to the characters.\n\n\n== Main characters ==\n\n\n=== Arthur Dent ===\nAlong with Ford Prefect, Arthur Philip Dent barely escapes the Earth\'s destruction as it is demolished to make way for a hyperspace bypass. Arthur spends the next several years, still wearing his dressing gown, helplessly launched from crisis to crisis while trying to straighten out his lifestyle. He rather enjoys tea, but seems to have trouble obtaining it in the far reaches of the galaxy. In time, he learns how to fly and carves a niche for himself as a sandwich-maker. He is also worried about "everything."\n\n\n=== Ford Prefect ===\nFord Prefect is Arthur Dent\'s friend. He also rescued Dent when the Earth is unexpectedly demolished to make way for a hyperspace bypass at the start of the story. Although his heart is in the right place and he is shown to be highly intelligent, resourceful, and even brave, Ford is essentially a dilettante when it comes to causes such as the search for the question to the ultimate answer of "life, the universe, and everything." Ford takes a more existential view of the universe, sometimes bordering on joyful nihilism. He is eccentric and broad-minded – no doubt due to his vast experience of roughing it around the galaxy – and has an off-key and often very dark sense of humour.\n\n\n=== Zaphod Beeblebrox ===\nZaphod Beeblebrox is a "semi-half-cousin" of Ford Prefect. He is hedonistic and irresponsible, narcissistic, and often extremely insensitive to the feelings of those around him. Zaphod invented the Pan Galactic Gargle Blaster. Zaphod wears unique clothing that contains a mixture of bright and contrasting colours to make him stand out and be the centre of attention wherever he goes. He was voted "Worst Dressed Sentient Being in the Known Universe" seven consecutive times. He\'s been described as "the best Bang since the Big One" by Eccentrica Gallumbits, and as "one hoopy frood" by others. He was briefly the President of the Galaxy and is the only man to have survived the Total Perspective Vortex (In the artificial galaxy created by Zarniwoop).\n\n\n=== Marvin the Paranoid Android ===\nMarvin the Paranoid Android is the ship\'s robot aboard the starship Heart of Gold. Built as one of many failed prototypes of Sirius Cybernetics Corporation\'s GPP (Genuine People Personalities) technology, Marvin is afflicted with severe depression and boredom, in part because he has a "brain the size of a planet" which he is seldom, if ever, given the chance to use. Indeed, the true horror of Marvin\'s existence is that no task he could be given would occupy even the tiniest fraction of his vast intellect. Marvin claims he is 50,000 times more intelligent than a human.\n\n\n=== Trillian ===\nTrillian (Tricia McMillan) is a mathematician and astrophysicist whom Arthur Dent attempted to talk to at a party in Islington. She and Arthur next meet six months later on the spaceship Heart of Gold, shortly after the Earth has been destroyed to make way for a hyperspace bypass.\n\n\n=== Slartibartfast ===\nSlartibartfast is a Magrathean, and a designer of planets. His favourite part of the job is creating coastlines, the most notable of which are the fjords found on the coast of Norway on planet Earth, for which he won an award. When Earth Mk. II is being made, Slartibartfast is assigned to the continent of Africa. He is unhappy about this because he has begun "doing it with fjords again" (arguing that they give a continent a lovely baroque feel), but has been told by his superiors that they are "not equatorial enough". In relation to this, he expresses the view that he would "far rather be happy than right any day."\n\n\n== Minor characters ==\n\n\n=== Agrajag ===\nAgrajag is a tragic and piteous creature who is continually reincarnated and subsequently killed, each time unknowingly, '),
  Document(metadata={'title': "The Hitchhiker's Guide to the Galaxy", 'summary': "The Hitchhiker's Guide to the Galaxy is a comedy science fiction franchise created by Douglas Adams. Originally a 1978 radio comedy broadcast on BBC Radio 4, it was later adapted to other formats, including novels, stage shows, comic books, a 1981 TV series, a 1984 text adventure game, and 2005 feature film.\nThe Hitchhiker's Guide to the Galaxy is an international multimedia phenomenon; the novels are the most widely distributed, having been translated into more than 30 languages by 2005. The first novel, The Hitchhiker's Guide to the Galaxy (1979), has been ranked fourth on the BBC's The Big Read poll. The sixth novel, And Another Thing..., was written by Eoin Colfer with additional unpublished material by Douglas Adams. In 2017, BBC Radio 4 announced a 40th-anniversary celebration with Dirk Maggs, one of the original producers, in charge. The first of six new episodes was broadcast on 8 March 2018.\nThe broad narrative of Hitchhiker follows the misadventures of the last surviving man, Arthur Dent, following the demolition of the Earth by a Vogon constructor fleet to make way for a hyperspace bypass. Dent is rescued from Earth's destruction by Ford Prefect—a human-like alien writer for the eccentric, electronic travel guide The Hitchhiker's Guide to the Galaxy—by hitchhiking onto a passing Vogon spacecraft. Following his rescue, Dent explores the galaxy with Prefect and encounters Trillian, another human who had been taken from Earth (before its destruction) by the self-centred President of the Galaxy Zaphod Beeblebrox and the depressed Marvin the Paranoid Android. Certain narrative details were changed among the various adaptations.", 'source': 'https://en.wikipedia.org/wiki/The_Hitchhiker%27s_Guide_to_the_Galaxy'}, page_content='The Hitchhiker\'s Guide to the Galaxy is a comedy science fiction franchise created by Douglas Adams. Originally a 1978 radio comedy broadcast on BBC Radio 4, it was later adapted to other formats, including novels, stage shows, comic books, a 1981 TV series, a 1984 text adventure game, and 2005 feature film.\nThe Hitchhiker\'s Guide to the Galaxy is an international multimedia phenomenon; the novels are the most widely distributed, having been translated into more than 30 languages by 2005. The first novel, The Hitchhiker\'s Guide to the Galaxy (1979), has been ranked fourth on the BBC\'s The Big Read poll. The sixth novel, And Another Thing..., was written by Eoin Colfer with additional unpublished material by Douglas Adams. In 2017, BBC Radio 4 announced a 40th-anniversary celebration with Dirk Maggs, one of the original producers, in charge. The first of six new episodes was broadcast on 8 March 2018.\nThe broad narrative of Hitchhiker follows the misadventures of the last surviving man, Arthur Dent, following the demolition of the Earth by a Vogon constructor fleet to make way for a hyperspace bypass. Dent is rescued from Earth\'s destruction by Ford Prefect—a human-like alien writer for the eccentric, electronic travel guide The Hitchhiker\'s Guide to the Galaxy—by hitchhiking onto a passing Vogon spacecraft. Following his rescue, Dent explores the galaxy with Prefect and encounters Trillian, another human who had been taken from Earth (before its destruction) by the self-centred President of the Galaxy Zaphod Beeblebrox and the depressed Marvin the Paranoid Android. Certain narrative details were changed among the various adaptations.\n\n\n== Spelling ==\nThe different versions of the series spell the title differently — thus Hitch-Hiker\'s Guide, Hitch Hiker\'s Guide, and Hitchhiker\'s Guide are used in different editions (UK or US), formats (audio or print), and compilations of the book, with some omitting the apostrophe. Some editions use different spellings on the spine and title page. The h2g2\'s English Usage in Approved Entries claims that Hitchhiker\'s Guide is the spelling that Adams preferred. At least two reference works make note of the inconsistency in the titles. Both, however, repeat the statement that Adams decided in 2000 that "everyone should spell it the same way [one word, no hyphen] from then on."\n\n\n== Synopsis ==\nThe various versions follow the same basic plot but they are in many places mutually contradictory, as Adams rewrote the story substantially for each new adaptation. Throughout all versions, the series follows the adventures of Arthur Dent, a hapless Englishman, following the destruction of the Earth by the Vogons (a race of unpleasant and bureaucratic aliens) to make way for a hyperspace bypass. Dent\'s adventures intersect with several other characters: Ford Prefect (an alien and researcher for the eponymous guidebook who rescues Dent from Earth\'s destruction), Zaphod Beeblebrox (Ford\'s eccentric semi-cousin and the Galactic President who has stolen the Heart of Gold, a spacecraft equipped with Infinite Improbability Drive), the depressed robot Marvin the Paranoid Android, and Trillian (formerly known as Tricia McMillan) who is a woman Arthur once met at a party in Islington and who—thanks to Beeblebrox\'s intervention—is the only other human survivor of Earth\'s destruction.\nIn their travels, Arthur comes to learn that the Earth was actually a giant supercomputer, created by another supercomputer, Deep Thought. Deep Thought had been built by its creators to give the answer to the "Ultimate Question of Life, the Universe, and Everything", which, after eons of calculations, was given simply as "42". Deep Thought was then instructed to design the Earth supercomputer to determine what the Question actually is. The Earth was subsequently destroyed by the Vogons moments before its calculations were completed, and Arthur becomes the target of the descendants of the Deep Thought creators, believing his mind must h')],
 'response': CitedAnswer(answer='42', citations=[1])}

In [45]:

result = await run("What is the exact number of grains of sand on all the beaches in the world?")

result

result = await run("What is the exact number of grains of sand on all the beaches in the world?") result

==============================
documents_retriever
==============================
==============================
context_checker
==============================
------------------------------
binary_score='no'
------------------------------
==============================
query_rewriter
==============================
------------------------------
How many grains of sand are on all beaches globally?
------------------------------
==============================
documents_retriever
==============================
==============================
context_checker
==============================
==============================
query_answerer
==============================
------------------------------
answer="I don't have enough information in the provided sources to answer this question." citations=[]
------------------------------
==============================
answer_checker
==============================
------------------------------
Pass Hallucination Check: no
------------------------------
==============================
retry
==============================
==============================
query_rewriter
==============================
------------------------------
How many grains of sand are on all beaches worldwide?
------------------------------
==============================
documents_retriever
==============================
==============================
context_checker
==============================
==============================
query_answerer
==============================
------------------------------
answer="I don't have enough information in the provided sources to answer this question." citations=[]
------------------------------
==============================
answer_checker
==============================
------------------------------
Pass Hallucination Check: no
------------------------------

Out[45]:

{'query': 'What is the exact number of grains of sand on all the beaches in the world?',
 'requery': 'How many grains of sand are on all beaches worldwide?',
 'can_requery': 0,
 'can_retry': 0,
 'documents': [Document(metadata={'title': 'Coastal erosion', 'summary': 'Coastal erosion is the loss or displacement of land, or the long-term removal of sediment and rocks along the coastline due to the action of waves, currents, tides, wind-driven water, waterborne ice, or other impacts of storms. The landward retreat of the shoreline can be measured and described over a temporal scale of tides, seasons, and other short-term cyclic processes. Coastal erosion may be caused by hydraulic action, abrasion, impact and corrosion by wind and water, and other forces, natural or unnatural.\nOn non-rocky coasts, coastal erosion results in rock formations in areas where the coastline contains rock layers or fracture zones with varying resistance to erosion. Softer areas become eroded much faster than harder ones, which typically result in landforms such as tunnels, bridges, columns, and pillars. Over time the coast generally evens out. The softer areas fill up with sediment eroded from hard areas, and rock formations are eroded away. Also erosion commonly happens in areas where there are strong winds, loose sand, and soft rocks. The blowing of millions of sharp sand grains creates a sandblasting effect. This effect helps to erode, smooth and polish rocks. The definition of erosion is grinding and wearing away of rock surfaces through the mechanical action of other rock or sand particles.\nAccording to the IPCC, sea level rise caused by climate change will increase coastal erosion worldwide, significantly changing the coasts and low-lying coastal areas.', 'source': 'https://en.wikipedia.org/wiki/Coastal_erosion'}, page_content="Coastal erosion is the loss or displacement of land, or the long-term removal of sediment and rocks along the coastline due to the action of waves, currents, tides, wind-driven water, waterborne ice, or other impacts of storms. The landward retreat of the shoreline can be measured and described over a temporal scale of tides, seasons, and other short-term cyclic processes. Coastal erosion may be caused by hydraulic action, abrasion, impact and corrosion by wind and water, and other forces, natural or unnatural.\nOn non-rocky coasts, coastal erosion results in rock formations in areas where the coastline contains rock layers or fracture zones with varying resistance to erosion. Softer areas become eroded much faster than harder ones, which typically result in landforms such as tunnels, bridges, columns, and pillars. Over time the coast generally evens out. The softer areas fill up with sediment eroded from hard areas, and rock formations are eroded away. Also erosion commonly happens in areas where there are strong winds, loose sand, and soft rocks. The blowing of millions of sharp sand grains creates a sandblasting effect. This effect helps to erode, smooth and polish rocks. The definition of erosion is grinding and wearing away of rock surfaces through the mechanical action of other rock or sand particles.\nAccording to the IPCC, sea level rise caused by climate change will increase coastal erosion worldwide, significantly changing the coasts and low-lying coastal areas.\n\n\n== Coastal processes ==\n\n\n=== Hydraulic action ===\nHydraulic action occurs when waves striking a cliff face compress air in cracks on the cliff face. This exerts pressure on the surrounding rock, and can progressively splinter and remove pieces. Over time, the cracks can grow, sometimes forming a cave. The splinters fall to the sea bed where they are subjected to further wave action.\n\n\n=== Attrition ===\nAttrition occurs when waves cause loose pieces of rock debris (scree) to collide with each other, grinding and chipping each other, progressively becoming smaller, smoother and rounder. Scree also collides with the base of the cliff face, chipping small pieces of rock from the cliff or have a corrasion (abrasion) effect, similar to sandpapering.\n\n\n=== Solution ===\nSolution is the process in which acids contained in sea water will dissolve some types of rock such as chalk or limestone.\n\n\n=== Abrasion ===\nAbrasion, also known as corrasion, occurs when waves break on cliff faces and slowly erode it. As the sea pounds cliff faces it also uses the scree from other wave actions to batter and break off pieces of rock from higher up the cliff face which can be used for this same wave action and attrition.\n\n\n=== Corrosion ===\nCorrosion or solution/chemical weathering occurs when the sea's pH (anything below pH 7.0) corrodes rocks on a cliff face. Limestone cliff faces, which have a moderately high pH, are particularly affected in this way. Wave action also increases the rate of reaction by removing the reacted material.\n\n\n== Factors that influence erosion rates ==\n\n\n=== Primary factors ===\n\nThe ability of waves to cause erosion of the cliff face depends on many factors.\nThe hardness (or inversely, the erodibility) of sea-facing rocks is controlled by the rock strength and the presence of fissures, fractures, and beds of non-cohesive materials such as silt and fine sand.\nThe rate at which cliff fall debris is removed from the foreshore depends on the power of the waves crossing the beach. This energy must reach a critical level to remove material from the debris lobe. Debris lobes can be very persistent and can take many years to completely disappear.\nBeaches dissipate wave energy on the foreshore and provide a measure of protection to the adjoining land.\nThe stability of the foreshore, or its resistance to lowering. Once stable, the foreshore should widen and become more effective at dissipating the wave energy, so that fewer and less powerful waves reach beyond it. Th"),
  Document(metadata={'title': 'Bentonite', 'summary': 'Bentonite ( BEN-tə-nyte) is an absorbent swelling clay consisting mostly of montmorillonite (a type of smectite) which can either be Na-montmorillonite or Ca-montmorillonite.  Na-montmorillonite has a considerably greater swelling capacity than Ca-montmorillonite.\nBentonite usually forms from the weathering of volcanic ash in seawater, or by hydrothermal circulation through the porosity of volcanic ash beds, which converts (devitrification) the volcanic glass (obsidian, rhyolite, dacite) present in the ash into clay minerals. In the mineral alteration process, a large fraction (up to 40–50 wt.%) of amorphous silica is dissolved and leached away, leaving the bentonite deposit in place. Bentonite beds are white or pale blue or green (traces of reduced Fe2+) in fresh exposures, turning to a cream color and then yellow, red, or brown (traces of oxidized Fe3+) as the exposure is weathered further.\nAs a swelling clay, bentonite has the ability to absorb large quantities of water, which increases its volume by up to a factor of eight. This makes bentonite beds unsuitable for building and road construction. However, the swelling property is used to advantage in drilling mud and groundwater sealants. The montmorillonite / smectite making up bentonite is an aluminium phyllosilicate mineral, which takes the form of microscopic platy grains. These give the clay a very large total surface area, making bentonite a valuable adsorbent. The plates also adhere to each other when wet. This gives the clay a cohesiveness that makes it useful as a binder and as an additive to improve the plasticity of kaolinite clay used for pottery.\nOne of the first findings of bentonite was in the Cretaceous Benton Shale near Rock River, Wyoming. The Fort Benton Group, along with others in stratigraphic succession, was named after Fort Benton, Montana, in the mid-19th century by Fielding Bradford Meek and F. V. Hayden of the U.S. Geological Survey. Bentonite has since been found in many other locations, including China and Greece (bentonite deposit of the Milos volcanic island in the Aegean Sea). The total worldwide production of bentonite in 2018 was 20,400,000 metric tons.\n\n', 'source': 'https://en.wikipedia.org/wiki/Bentonite'}, page_content='Bentonite ( BEN-tə-nyte) is an absorbent swelling clay consisting mostly of montmorillonite (a type of smectite) which can either be Na-montmorillonite or Ca-montmorillonite.  Na-montmorillonite has a considerably greater swelling capacity than Ca-montmorillonite.\nBentonite usually forms from the weathering of volcanic ash in seawater, or by hydrothermal circulation through the porosity of volcanic ash beds, which converts (devitrification) the volcanic glass (obsidian, rhyolite, dacite) present in the ash into clay minerals. In the mineral alteration process, a large fraction (up to 40–50 wt.%) of amorphous silica is dissolved and leached away, leaving the bentonite deposit in place. Bentonite beds are white or pale blue or green (traces of reduced Fe2+) in fresh exposures, turning to a cream color and then yellow, red, or brown (traces of oxidized Fe3+) as the exposure is weathered further.\nAs a swelling clay, bentonite has the ability to absorb large quantities of water, which increases its volume by up to a factor of eight. This makes bentonite beds unsuitable for building and road construction. However, the swelling property is used to advantage in drilling mud and groundwater sealants. The montmorillonite / smectite making up bentonite is an aluminium phyllosilicate mineral, which takes the form of microscopic platy grains. These give the clay a very large total surface area, making bentonite a valuable adsorbent. The plates also adhere to each other when wet. This gives the clay a cohesiveness that makes it useful as a binder and as an additive to improve the plasticity of kaolinite clay used for pottery.\nOne of the first findings of bentonite was in the Cretaceous Benton Shale near Rock River, Wyoming. The Fort Benton Group, along with others in stratigraphic succession, was named after Fort Benton, Montana, in the mid-19th century by Fielding Bradford Meek and F. V. Hayden of the U.S. Geological Survey. Bentonite has since been found in many other locations, including China and Greece (bentonite deposit of the Milos volcanic island in the Aegean Sea). The total worldwide production of bentonite in 2018 was 20,400,000 metric tons.\n\n\n== Types ==\n\nIn geology, the term bentonite is applied to a type of claystone (a clay rock, not a clay mineral) composed mostly of montmorillonite (a clay mineral from the smectite group). It forms by devitrification of volcanic ash or tuff, typically in a marine environment. This results in a very soft, porous rock that may contain residual crystals of more resistant minerals, and which feels soapy or greasy to the touch. However, in commercial and industrial applications, the term bentonite is used more generally to refer to any swelling clay composed mostly of smectite clay minerals, which includes montmorillonite. The undifferentiated reference to the weathered volcanic rock for the geologist or to the industrial mixture of swelling clays can be a source of confusion.\nThe montmorillonite making up bentonite is an aluminium phyllosilicate mineral whose crystal structure is described as low-charge TOT. This means that a crystal of montmorillonite consists of layers, each of which is made up of two T sheets bonded to either side of an O sheet. The T sheets are so called because each aluminium or silicon ion in the sheet is surrounded by four oxygen ions arranged as a tetrahedron. The O sheets are so called because each aluminium ion is surrounded by six oxygen or hydroxyl ions arranged as an octahedron. The complete TOT layer has a weak negative electrical charge, and this is neutralized by calcium or sodium cations that bind adjacent layers together, with a distance between layers of about 1 nanometer. Because the negative charge is weak, only a fraction of the possible cation sites on the surface of a TOT layer actually contain calcium or sodium. Water molecules can easily infiltrate between sheets and fill the remaining sites. This accounts for the swelling property of montmorillonite a'),
  Document(metadata={'title': 'Marine habitat', 'summary': 'A marine habitat is a habitat that supports marine life. Marine life depends in some way on the saltwater that is in the sea (the term marine comes from the Latin mare, meaning sea or ocean). A habitat is an ecological or environmental area inhabited by one or more living  species. The marine environment supports many kinds of these habitats. \nMarine habitats can be divided into coastal and open ocean habitats. Coastal habitats are found in the area that extends from as far as the tide comes in on the shoreline out to the edge of the continental shelf. Most marine life is found in coastal habitats, even though the shelf area occupies only seven percent of the total ocean area. Open ocean habitats are found in the deep ocean beyond the edge of the continental shelf.\nAlternatively, marine habitats can be divided into pelagic and demersal zones. Pelagic habitats are found near the surface or in the open water column, away from the bottom of the ocean. Demersal habitats are near or on the bottom of the ocean. An organism living in a pelagic habitat is said to be a pelagic organism, as in pelagic fish. Similarly, an organism living in a demersal habitat is said to be a demersal organism, as in demersal fish. Pelagic habitats are intrinsically shifting and ephemeral, depending on what ocean currents are doing.\nMarine habitats can be modified by their inhabitants. Some marine organisms, like corals, kelp, mangroves and seagrasses, are ecosystem engineers which reshape the marine environment to the point where they create further habitat for other organisms. By volume the ocean provides most of the habitable space on the planet.\n\n', 'source': 'https://en.wikipedia.org/wiki/Marine_habitat'}, page_content="A marine habitat is a habitat that supports marine life. Marine life depends in some way on the saltwater that is in the sea (the term marine comes from the Latin mare, meaning sea or ocean). A habitat is an ecological or environmental area inhabited by one or more living  species. The marine environment supports many kinds of these habitats. \nMarine habitats can be divided into coastal and open ocean habitats. Coastal habitats are found in the area that extends from as far as the tide comes in on the shoreline out to the edge of the continental shelf. Most marine life is found in coastal habitats, even though the shelf area occupies only seven percent of the total ocean area. Open ocean habitats are found in the deep ocean beyond the edge of the continental shelf.\nAlternatively, marine habitats can be divided into pelagic and demersal zones. Pelagic habitats are found near the surface or in the open water column, away from the bottom of the ocean. Demersal habitats are near or on the bottom of the ocean. An organism living in a pelagic habitat is said to be a pelagic organism, as in pelagic fish. Similarly, an organism living in a demersal habitat is said to be a demersal organism, as in demersal fish. Pelagic habitats are intrinsically shifting and ephemeral, depending on what ocean currents are doing.\nMarine habitats can be modified by their inhabitants. Some marine organisms, like corals, kelp, mangroves and seagrasses, are ecosystem engineers which reshape the marine environment to the point where they create further habitat for other organisms. By volume the ocean provides most of the habitable space on the planet.\n\n\n== Overview ==\n\nIn contrast to terrestrial habitats, marine habitats are shifting and ephemeral. Swimming organisms find areas by the edge of a continental shelf a good habitat, but only while upwellings bring nutrient rich water to the surface. Shellfish find habitat on sandy beaches, but storms, tides and currents mean their habitat continually reinvents itself.\nThe presence of seawater is common to all marine habitats. Beyond that many other things determine whether a marine area makes a good habitat and the type of habitat it makes. For example:\n\ntemperature – is affected by geographical latitude, ocean currents, weather, the discharge of rivers, and by the presence of hydrothermal vents or cold seeps\nsunlight – photosynthetic processes depend on how deep and turbid the water is\nnutrients – are transported by ocean currents to different marine habitats from land runoff, or by upwellings from the deep sea, or they sink through the sea as marine snow\nsalinity – varies, particularly in estuaries or near river deltas, or by hydrothermal vents\ndissolved gases – oxygen levels in particular, can be increased by wave actions and decreased during algal blooms\nacidity – this is partly to do with dissolved gases above, since the acidity of the ocean is largely controlled by how much carbon dioxide is in the water.\nturbulence –  ocean waves, fast currents and the agitation of water affect the nature of habitats\ncover – the availability of cover such as the adjacency of the sea bottom, or the presence of floating objects\nsubstrate – The slope, orientation, profile and rugosity of hard substrates, and particle size, sorting and density of unconsolidated sediment bottoms can make a big difference to the life forms that can settle on it.\nthe occupying organisms themselves – since organisms modify their habitats by the act of occupying them, and some, like corals, kelp, mangroves and seagrasses, create further habitats for other organisms.\n\nThere are five major oceans, of which the Pacific Ocean is nearly as large as the rest put together. Coastlines fringe the land for nearly 380,000 kilometres.\n\nAltogether, the ocean occupies 71 percent of the world surface, averaging nearly four kilometres in depth. By volume, the ocean contains more than 99 percent of the Earth's liquid water. The science fiction writer Arthur C. Cla")],
 'response': CitedAnswer(answer="I don't have enough information in the provided sources to answer this question.", citations=[]),
 'is_fail': True}

In [ ]:

Comments