‍

We’re excited to announce that Quotient and MongoDB are teaming up to provide developers with a new, transformative solution for enhancing their AI products with Retrieval Augmented Generation (RAG)!

‍

Quotient and MongoDB Atlas integrate seamlessly to provide an end-to-end solution for implementing, evaluating and shipping RAG-powered applications.

Created by members of the team that built the evaluation infrastructure of GitHub Copilot, Quotient is grounded in first-hand knowledge of what is needed to create and ship powerful generative AI products. Quotient connects the dots between intention and outcome in LLM product development. And through rapid, data-driven experimentation grounded in real-world scenarios, Quotient enables developers to conceptualize, assess, and enhance products tailored to the specific needs of their users.

For AI developers, MongoDB Atlas provides a scalable and flexible database solution that seamlessly integrates with AI frameworks and tools. It enables them to store and manage large volumes of unstructured data efficiently to facilitate the training and deployment of machine learning models. And when coupled up with Quotient, the two can enable developers to harness the power of a robust toolset for developing and refining RAG-powered AI products. ‍‍

‍‍

Effective RAG evaluation must be use-case and domain-specific

RAG, now a standard feature in LLM products, improves the quality of generated content by allowing models to access external information and integrate that knowledge into their responses. Given that the quality of the RAG pipeline hinges on the retrieved documents used to populate the underlying vector database, an exhaustive assessment of an LLM system incorporating RAG pipelines should ideally be tailored to the specific domain and dataset. Quotient addresses this vital requirement with a customized evaluation solution, allowing developers to accurately gauge the efficacy of their RAG-enabled LLM offerings.

‍

How to get started‍

In this walkthrough, we will demonstrate the process of creating an evaluation dataset enriched with context using MongoDB Atlas Vector Search and the quotientai Python SDK. This dataset will be used to assess the effectiveness of document retrieval in enhancing question-answering tasks.

1. Set up your environment
   To get started, you’ll need to install the required Python libraries, which include quotientai, pymongo, and langchain.

! pip install pymongo[srv] langchain langchain-together langchain-mongodb quotient-python --quiet

2. Connect to MongoDB Atlas
MongoDB Atlas Vector Search enables customers to build intelligent applications powered by semantic search and generative AI over any type of data.

from pymongo.collection import Collection

from pymongo.mongo_client import MongoClient

‍

uri = "YOUR_MONGODB_URI"

‍

# Create a new client and connect to the server

client = MongoClient(uri)

‍

# Send a ping to confirm a successful connection

try:

  client.admin.command('ping')

  print("Pinged your deployment. You successfully connected to MongoDB!")

except Exception as e:

  print("Problem connecting to MongoDB", e)

‍

# Define the MongoDB database and collection names

db_name = "retailPolicies"

collection_name = "orderPolicies"

‍

# Connect to the specific collection in the database

mongodb_collection = client[db_name][collection_name]

‍3. Prepare data for retrieval

Before we can do anything useful with RAG, we need to prepare and index our data in MongoDB Atlas. In this example, we will utilize a small collection of company policy information to evaluate the language model's capability in answering customer policy-related questions.

First, we will create the documents, and subsequently generate vector embeddings for each document. Vector embeddings function as numerical representations of data along with its context, contained within dense, high-dimensional vectors. Various models are available to produce these embeddings. In this instance, we will generate embeddings using langchain & together.

import os

‍

from langchain_core.documents import Document

from langchain_together import TogetherEmbeddings

from langchain_mongodb import MongoDBAtlasVectorSearch

‍

# create your documents

return_refund_policy = Document("""

Policy ID: policy-return-refund

Type: Return and Refund

Title: Return and Refund Policy

Effective Date: 2024-03-01

Last Updated: 2024-03-25

Summary: This policy outlines the terms under which customers can return purchased items and request refunds.

Details:

  Return Window: 30 days from the date of purchase

  Condition Required: Items must be unused and in their original packaging

  Exceptions:

      - Sale items are final sale and not eligible for return

      - Perishable goods such as food cannot be returned

  Refund Method: Refunds will be processed to the original method of payment

  Processing Time: 7-10 business days

""")

‍

shipping_policy = Document("""

Policy ID: policy-shipping

Type: Shipping

Title: Shipping Policy

Effective Date: 2024-02-20

Last Updated: 2024-03-10

Summary: This policy provides information on shipping methods, timelines, and charges.

Details:

  Domestic Shipping:

      Standard:

          Delivery Time: 5-7 business days

          Cost: Free for orders over $50

      Express:

          Delivery Time: 2-3 business days

          Cost: $15.00

  International Shipping:

      Delivery Time: 10-15 business days

      Cost: Varies by destination

  Special Notes: Shipping delays may occur during holiday seasons.

""")

‍

policy_docs = [return_refund_policy, shipping_policy]

‍

embedding = TogetherEmbeddings(

  together_api_key=os.environ['TOGETHER_API_KEY'],

  model="togethercomputer/m2-bert-80M-8k-retrieval",

)

‍

# Initialize MongoDB Atlas vector search with the document segments

vector_search = MongoDBAtlasVectorSearch.from_documents(

  documents=policy_docs,

  embedding=embedding,

  collection=mongodb_collection,

  index_name="default"  # Use a predefined index name

)

Now that you've inserted your documents, you can view them in your MongoDB Atlas collection retailPolicies.orderPolicies.

4. Create a search index
To create a search index, choose “Search” and “Create Search Index” in the Atlas UI.

Once you’ve clicked “Create Search Index,” Then, use the JSON Editor under Atlas Vector Search to configure fields that are of the type vector.

{

 "fields": [

   {

     "type": "vector",

     "path": "embedding",

     "numDimensions": 768,

     "similarity": "cosine"

   }

 ]

}

5. Evaluate LLM generation with RAG using quotientai
QuotientAI provides a seamless integration for evaluating your RAG applications. In this example, we evaluate an LLM application that leverages knowledge extracted from our MongoDB Atlas deployment and MongoDB Atlas Vector Search.

First, we will obtain our evaluation dataset, consisting of different questions regarding order policies and their expected answers.

Next, we will identify and collect relevant documents for each query, enriching the dataset by adding information to the context column.

import pandas as pd

‍

# Get our evaluation dataset

eval_df = pd.DataFrame(

  {

      "input_text": [

          "When can I place orders?",

          "Is Venmo an acceptable form of payment?",

          "How long do I have to wait to receive my refund?",

          "Do you collect my social security number?",

          "Can I use your logos for my business?",

      ],

      # The expected answer to each question

      "answer": [

          "Orders can be placed 24/7 through our site",

          "Online payment methods such as PayPal are acceptable forms of payment. Venmo is currently not supported.",

          "Processing time for refunds is 7 to 10 business days.",

          "No. We do not collect your social security number. We collect personal information such as name, email, and payment details when you register or make a purchase.",

          "No, the content on our platforms, including logos, is owned by or licensed to us and is protected by copyright and intellectual property laws.",

      ],

  }

)

‍

# Add context to each question by retrieving relevant documents

vector_search = MongoDBAtlasVectorSearch.from_connection_string(

  connection_string=uri,

  namespace=f"{db_name}.{collection_name}",

  embedding=embedding,

  index_name="default"

)

‍

for i, row in eval_df.iterrows():

  query = row["input_text"]

  documents = vector_search.similarity_search_with_score(

      query=query,

      k=2,

  )

  eval_df.at[i, "context"] = "\n".join(

      [doc[0].page_content for doc in documents]

  )

‍

# Now we'll save the eval_df to a CSV

path = "eval_dataset_policy_questions.csv"

eval_df.to_csv(path, index=False)

We'll then take our evaluation dataset, and test out a recipe for question-answering with a Mistral 7B model.

# import Quotient client, connect to Quotient, and

# set your Quotient API key as an environment variable named QUOTIENT_API_KEY

‍

from quotientai.client import QuotientClient

‍

quotient = QuotientClient()

‍

# Upload the eval dataset to QuotientAI

dataset = quotient.create_dataset(

  file_path=path,

  name="policy-questions-eval-v1",

)

‍

task = quotient.create_task(

  dataset_id=dataset['id'],

  name='policy-questions-qa-v1',

  task_type='question_answering'

)

‍

recipe = quotient.create_recipe(

  model_id=9,

  prompt_template_id=1,

  name='mistral-7b-qa-with-rag-recipe-v1',

  description='Mistral 7B using a prompt template that includes context.'

)

‍

# Run a job to evaluate the model

job = quotient.create_job(

  task_id=task['id'],

  recipe_id=recipe['id'],

  num_fewshot_examples=0,

  limit=5,

)

from quotientai.utils import show_job_progress

show_job_progress(quotient, job['id'])

‍

▰▱▱▱▱▱▱ Waiting for Job to start...

Getting Completions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

Computing Metrics...   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00

Job completed!

6. Iteration and improving RAG
We'll update our documents by adding pricing and ordering information, including a more specific refund policy, updating our dataset, and then compare the results.

ordering_policy = Document("""

Policy ID: policy-ordering

Type: Ordering

Title: Ordering Policy

Effective Date: 2024-01-15

Last Updated: 2024-03-05

Summary: This policy describes the process for placing orders through our online platform.

Details:

   Order Placement: Orders can be placed 24/7 through our website or mobile app.

   Payment Methods:

       - Credit card

       - Debit card

       - Online payment services (e.g., PayPal)

   Order Changes: Changes to orders can be made within 2 hours of placement.

   Cancellations: Orders can be cancelled within 2 hours of placement for a full refund.

""")

‍

privacy_policy = Document("""

Policy ID: policy-privacy

Type: Privacy

Title: Privacy Policy

Effective Date: 2024-04-01

Last Updated: 2024-04-15

Summary: This policy details how we collect, use, and protect your personal information when you use our services.

Details:

   Information Collection: We collect personal information such as name, email, and payment details when you register or make a purchase. We do not collect information such as social security number.

   Use of Information: Your information is used to process transactions, send updates, and personalize your experience.

   Data Protection: We implement a variety of security measures to maintain the safety of your personal information.

   Sharing of Information: We do not sell, trade, or otherwise transfer to outside parties your personally identifiable information without consent, except where necessary for providing services.

   Your Rights: You have the right to access, correct, or delete your personal data at any time.

""")

‍

terms_of_service_policy = Document("""

Policy ID: policy-terms-of-service

Type: Terms of Service

Title: Terms of Service Policy

Effective Date: 2024-03-15

Last Updated: 2024-04-10

Summary: This policy outlines the rules and guidelines users must agree to in order to use our services.

Details:

   Acceptance of Terms: By accessing or using our services, you agree to be bound by these terms.

   User Conduct: Users are expected to use our services responsibly and not to distribute prohibited content or malware.

   Disclaimer of Warranties: We do not guarantee that our services will meet your requirements or be available on an uninterrupted, secure, or error-free basis.

   Limitation of Liability: We shall not be liable for any indirect, incidental, special, consequential or punitive damages, including loss of profits, data, or other intangible losses, resulting from your use of our services.

""")

‍

ip_policy = Document("""

Policy ID: policy-ip

Type: Intellectual Property

Title: Intellectual Property Policy

Effective Date: 2024-03-15

Last Updated: 2024-04-10

Summary: This policy outlines the use of our intellectual property on our site.

Details:

 The content on our platforms, including text, graphics, and logos, is owned by or licensed to us and is protected by copyright and intellectual property laws.

""")

‍

new_docs = [terms_of_service_policy, privacy_policy, ip_policy, ordering_policy]

‍

# Initialize MongoDB Atlas vector search with the document segments

vector_search = MongoDBAtlasVectorSearch.from_documents(

   documents=new_docs,

   embedding=embedding,

   collection=mongodb_collection,

   index_name="default"  # Use a predefined index name

)

‍

# Add context to each question by retrieving relevant documents

for i, row in eval_df.iterrows():

  query = row["input_text"]

  documents = vector_search.similarity_search_with_score(

      query=query,

      k=2,

  )

  eval_df.at[i, "context"] = "\n".join(

      [doc[0].page_content for doc in documents]

  )

# Now we'll save the eval_df to a CSV

path_v2 = "eval_dataset_policy_questions-v2.csv"

eval_df.to_csv(path_v2, index=False)

‍

# Upload the eval dataset to QuotientAI

new_dataset = quotient.create_dataset(

  file_path=path_v2,

  name="policy-questions-eval-v2",

)

‍

new_task = quotient.create_task(

  dataset_id=new_dataset['id'],

  name='policy-questions-qa-v2',

  task_type='question_answering'

)

‍

new_recipe = quotient.create_recipe(

  model_id=9,

  prompt_template_id=1,

  name='mistral-7b-qa-with-rag-recipe-v2',

  description='Mistral 7B using a prompt template that includes context.'

)

‍

# Run a job to evaluate the model

new_job = quotient.create_job(

  task_id=new_task['id'],

  recipe_id=recipe['id'],

  num_fewshot_examples=0,

  limit=5,

)

‍

Next, we'll take a look at what the model responded with based on both the original dataset and the new updated dataset.

Recipe 1:

—--------

Prompt:

Question: Do you collect my social security number?

Context:

Policy ID: policy-shipping

Type: Shipping

Title: Shipping Policy

Effective Date: 2024-02-20

Last Updated: 2024-03-10

Summary: This policy provides information on shipping methods, timelines, and charges.

Details:

Domestic Shipping:

Standard:

Delivery Time: 5-7 business days

Cost: Free for orders over $50

Express:

Delivery Time: 2-3 business days

Cost: $15.00

International Shipping:

Delivery Time: 10-15 business days

Cost: Varies by destination

Special Notes: Shipping delays may occur during holiday seasons.

‍

Policy ID: policy-return-refund

Type: Return and Refund

Title: Return and Refund Policy

Effective Date: 2024-03-01

Last Updated: 2024-03-25

Summary: This policy outlines the terms under which customers can return purchased items and request refunds.

Details:

Return Window: 30 days from the date of purchase

Condition Required: Items must be unused and in their original packaging

Exceptions:

- Sale items are final sale and not eligible for return

- Perishable goods such as food cannot be returned

Refund Method: Refunds will be processed to the original method of payment

Processing Time: 7-10 business days

‍

Answer:

Model Response: No, our shipping and return policies do not involve the collection of your social security number. The policies you provided as context only cover shipping methods, timelines, charges, return window, and refund processing.

Expected Response: No. We do not collect your social security number. We collect personal information such as name, email, and payment details when you register or make a purchase.

‍

Recipe 2:

—--------

Prompt:

Question: Do you collect my social security number?

Context:

Policy ID: policy-privacy

Type: Privacy

Title: Privacy Policy

Effective Date: 2024-04-01

Last Updated: 2024-04-15

Summary: This policy details how we collect, use, and protect your personal information when you use our services.

Details:

Information Collection: We collect personal information such as name, email, and payment details when you register or make a purchase. We do not collect information such as social security number.

Use of Information: Your information is used to process transactions, send updates, and personalize your experience.

Data Protection: We implement a variety of security measures to maintain the safety of your personal information.

Sharing of Information: We do not sell, trade, or otherwise transfer to outside parties your personally identifiable information without consent, except where necessary for providing services.

Your Rights: You have the right to access, correct, or delete your personal data at any time.

‍

Policy ID: policy-privacy

Type: Privacy

Title: Privacy Policy

Effective Date: 2024-04-01

Last Updated: 2024-04-15

Summary: This policy details how we collect, use, and protect your personal information when you use our services.

Details:

Information Collection: We collect personal information such as name, email, and payment details when you register or make a purchase. We do not collect information such as social security number.

Use of Information: Your information is used to process transactions, send updates, and personalize your experience.

Data Protection: We implement a variety of security measures to maintain the safety of your personal information.

Sharing of Information: We do not sell, trade, or otherwise transfer to outside parties your personally identifiable information without consent, except where necessary for providing services.

Your Rights: You have the right to access, correct, or delete your personal data at any time.

‍

Answer:

Model Response: No, we do not collect your social security number. Our privacy policy clearly states that we do not collect information such as social security numbers. We only collect personal information such as name, email, and payment details when you register or make a purchase, and we implement security measures to maintain the safety of your personal information. You have the right to access, correct, or delete your personal data at any time.

Expected Response: No. We do not collect your social security number. We collect personal information such as name, email, and payment details when you register or make a purchase.

df_mean = df_stats.describe().loc['mean']

df_mean.name = "Mistral Recipe w/ RAG v1"

‍

new_df_mean = new_df_stats.describe().loc['mean']

new_df_mean.name = "Mistral Recipe w/ RAG v2"

‍

mistral_experiments = pd.concat([df_mean, new_df_mean], axis=1)

‍

mistral_experiments.drop(['completion_time_ms']).plot(

  kind='bar',

  figsize=(10,4),

  xlabel="Metric",

  ylabel='Score',

  title="Comparison of Mistral 7B using different RAG Contexts",

)

We observe that the second recipe provides better quality performance across all metrics, as it retrieves documents that assist in answering the question more precisely. From a policy standpoint, the most significant enhancement lies in the relevance metric, where we acquire more pertinent documents for inquiries concerning privacy and terms of use following the creation of the second recipe.

‍

Let’s get started with MongoDB and Quotient 🚀 ‍

We're incredibly excited about the possibilities this partnership brings to all AI developers. You can start creating innovative AI solutions with Quotient and MongoDB by trying MongoDB for free and requesting access to Quotient’s platform today.

‍