Empowering Your AI Agents: Integrating Internet Search with AutoGen and Google Search API

Have you ever felt that your AI assistant was…out of touch? Perhaps stuck in a pre-internet era? Imagine, instead, an AI that doesn't just converse; it actively seeks out the most current information – transforming from a mere conversational partner into a powerful research tool. Today, we explore that very possibility.

We will delve into how to enhance your AI agents using the AutoGen framework and the reliable Google Search API. We will construct an AI system capable of accessing the vast resources of the internet. This will allow your AI assistants to be not only adept at conversation, but also remarkably resourceful. This guide provides a straightforward walkthrough, without any confusing jargon, ensuring that even those new to AI development can create their own intelligent tool.

Ready to explore the possibilities? Let's begin!

Why Equip Your AI with Real-Time Data? The Power of RAG

AI agents, particularly those powered by Large Language Models (LLMs), excel at generating text, mastering multiple languages, and engaging in captivating dialogues. However, their knowledge is often restricted to their initial training data, which can quickly become outdated. This is where Retrieval-Augmented Generation (RAG) plays a crucial role.

RAG allows your AI to access external knowledge sources during the generation process. In this project, we will leverage the Google Search API, a powerful tool that enables our AI to retrieve real-time information from the internet. By integrating this capability into our systems, we are transforming our AIs into dynamic and insightful entities. Consider this: humans don't simply rely on memorized facts; they actively seek out new information to make informed decisions - our enhanced AI Agent will accomplish the same.

Setting Up Your Development Environment

Before we proceed to the more exciting aspects, we need to ensure our development environment is properly configured. Here's what you'll need:

1. Python Installation: Ensure you have Python 3.11 or a later version installed. Run python --version in your terminal to confirm.

2. Virtual Environment: Create a virtual environment using:

   python3.11 -m venv agentvenv 

   Activate your python virtual environment using:

   source ~/agentvenv/bin/activate

3. Required Libraries: Install the necessary packages by executing:

   pip install autogen requests bs4 python-dotenv pydantic

   These packages include the AutoGen framework, bs4 (for web scraping), dotenv (for managing API keys), and pydantic (for data validation).

4. Google Search API Keys: Visit the Google Developers Console to obtain your Programmable Search API key. You will need to create a search engine, get the API key, and note your Search engine ID. Follow below steps:

   1. Go to Google Developers, use Programmable Search Engine, and log on or create account.

   2. Go to control panel and click Add button

   3. Enter a search engine name, set the other properties to suit your needs, verify you're not a robot and click Create button.

   4. Generate API key and get the Search engine ID. (Available after the engine is created)

5. Project Directory: Create a directory called autogen-google-tool. Inside, create a .env file to store your API keys:

   GOOGLE_API_KEY=<your_google_api_key>
   GOOGLE_SE_ID=<your_search_engine_id>

Finally, create a file named agent.py within this folder. Now, you're ready for the coding phase.

Building Your Search-Enabled AI Agent: Step-by-Step

Let's follow these steps to create an AI agent that can leverage the internet through the Google Search API:

1. Library Imports: Begin by adding the following at the top of your agent.py file:

   import os, time, requests 
   from autogen import ConversableAgent 
   from dotenv import load_dotenv 
   from bs4 import BeautifulSoup  
   
   #load your environment variables
   load_dotenv() 

   This code imports the necessary libraries to interact with the Google Search API, create AutoGen agents, and load environment variables.

2. Language Model Configuration:

   llm_config = {
       "config_list": [
           {
               "model": "mistral",
               "base_url": "http://localhost:11434/v1",
               "api_key": "ollama",
               "cache_seed": None,
               "price": [0.0, 0.0],
           }
       ],
       "temperature": 0,
   }

   Here, we are setting up the language model for our agent. This configuration is tailored for the Mistral model, hosted locally via Ollama, allowing for controlled model execution.

3. Creating Conversational Agents:

   assistant = ConversableAgent(
       name="Assistant",
       system_message="You are a helpful AI assistant with access to internet search capabilities.",
       llm_config=llm_config,
   )
   
   user_proxy = ConversableAgent(
       name="User",
       human_input_mode="NEVER",
       llm_config=False
   )

   We've constructed two key agents: an assistant agent, which handles user queries and utilizes internet search, and a user_proxy agent, which executes search functions without a language model.

4. Implement and Register the Search Function:

   @assistant.register_for_llm(
       name="web_search",
       description="Searches the web according to a given query",
   )
   @user_proxy.register_for_execution(name="web_search")
   def google_search(query: str, num_results: int = 2, max_chars: int = 500) -> list:
        api_key = os.getenv("GOOGLE_API_KEY")
        search_engine_id = os.getenv("GOOGLE_SE_ID")
   
        url = "https://customsearch.googleapis.com/customsearch/v1"
        params = {"key": str(api_key), "cx": str(search_engine_id), "q": str(query), "num": str(num_results)}
   
        response = requests.get(url, params=params)
        if response.status_code != 200:
            print(response.json())
            raise Exception(f"Error in API request: {response.status_code}")
   
        results = response.json().get("items", [])
   
        def get_page_content(url: str) -> str:
           try:
               response = requests.get(url, timeout=10)
               soup = BeautifulSoup(response.content, "html.parser")
               text = soup.get_text(separator=" ", strip=True)
               words = text.split()
               content = ""
               for word in words:
                  if len(content) + len(word) + 1 > max_chars:
                       break
                  content += " " + word
               return content.strip()
           except Exception as e:
               print(f"Error fetching {url}: {str(e)}")
               return ""
   
        enriched_results = []
        for item in results:
          body = get_page_content(item["link"])
          enriched_results.append(
           {"title": item["title"], "link": item["link"], "snippet": item["snippet"], "body": body}
          )
          time.sleep(1)  # Be respectful to servers
   
        return enriched_results

   This function is the core search mechanism. It uses the Google Search API, formats the results, and registers it with AutoGen. While the assistant initiates the call, user_proxy executes the function, enhancing security and code organization.

5. Main Chat Loop:

   def main():
       while True:
           user_input = input("User: ")
   
           if user_input.lower() in ["exit", "quit", "bye"]:
               print("Chatbot: Goodbye! Have a great day!")
               break
   
           chat_result = user_proxy.initiate_chat(
               assistant,
               message=user_input,
               max_turns=2,
           )
   
           reply = next(
              (
               msg["content"]
                   for msg in chat_result.chat_history
                  if msg.get("name") == "Assistant"
               ),
               "I apologize, but I couldn't generate a response.",
          )
          print(f"Chatbot: {reply}")

   This function manages the interactive dialogue: the user provides input, the AI responds, and the process repeats until a termination command is given.

6. Run the Chatbot:

   if __name__ == "__main__":
       main()

   Now, run your script by entering python agent.py and start asking questions. The search tool will be used as necessary to provide up-to-date information.

Conclusion: The Rise of Intelligent, Resourceful AI

Congratulations! You have successfully developed an AI agent that is not only conversational, but also adept at utilizing the internet to provide real-time, fact-checked responses. This integration unlocks a wide array of applications.

• Real-Time Information: No longer constrained by static data, your AI can access the constantly updated web.

• Contextual Understanding: The AI can deliver well-researched answers based on information sourced directly from the internet.

• Adaptive Knowledge: Continual information processing allows the AI to adapt to changes swiftly.

Imagine AI assistants that aid in research, keep you informed about current events, or plan complex trips. The potential applications are vast. This post should serve as a stepping stone toward more advanced AI systems, combining LLMs with powerful external tools.

Next Steps for Innovation

• Experiment with different types of queries and observe how the AI Agent responds.

• Consider how this technology could be applied to your specific projects.

• Share your thoughts and experiences in the comments. Your insights are highly valued.

• Explore further by integrating other external APIs.

The future of AI is one of dynamic collaboration, where AI is not just intelligent, but also incredibly resourceful. Let’s continue building this future together.