ScrapeWebsiteTool - SSLCertVerificationError

pratikchhapolika · February 5, 2025, 4:18pm

Description

I am trying to use ScrapeWebsiteTool tools on my jupyter notebook, giving SSLCertVerificationError, but simple requests works.

**crewai==0.28.8

crewai_tools==0.1.6

langchain_community==0.0.29**

Steps to Reproduce

utils.py

import os
from dotenv import load_dotenv, find_dotenv

# these expect to find a .env file at the directory above the lesson.                                                                                                                     # the format for that file is (without the comment)                                                                                                                                       #API_KEYNAME=AStringThatIsTheLongAPIKeyFromSomeService
def load_env():
    _ = load_dotenv(find_dotenv())


def get_serper_api_key():
    load_env()
    openai_api_key = os.getenv("SERPER_API_KEY")
    return openai_api_key

import os
from utils import get_serper_api_key

from crewai_tools import ScrapeWebsiteTool

# To enable scrapping any website it finds during it's execution
tool = ScrapeWebsiteTool(website_url="https://docs.crewai.com/how-to/Creating-a-Crew-and-kick-it-off/")

# Extract the text from the site
text = tool.run()
print(text)

Does this automatically get the get_serper_api_key, I don’t see I am using it anywhere in code?

ERROR:

SSLCertVerificationError                  Traceback (most recent call last)
File [~/Library/Python/3.12/lib/python/site-packages/urllib3/connectionpool.py:466](http://localhost:8888/lab/tree/Downloads/inference_cgs/crewai/~/Library/Python/3.12/lib/python/site-packages/urllib3/connectionpool.py#line=465), in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    465 try:
--> 466     self._validate_conn(conn)
    467 except (SocketTimeout, BaseSSLError) as e:

SSLError: HTTPSConnectionPool(host='docs.crewai.com', port=443): Max retries exceeded with url: [/how-to/Creating-a-Crew-and-kick-it-off/](http://localhost:8888/how-to/Creating-a-Crew-and-kick-it-off/) (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))

Simle request works

 import requests
response = requests.get("https://docs.crewai.com/how-to/Creating-a-Crew-and-kick-it-off/", verify=False)
print(response.text)

Expected behavior

Give text output

Simle request works

 import requests
response = requests.get("https://docs.crewai.com/how-to/Creating-a-Crew-and-kick-it-off/", verify=False)
print(response.text)

Screenshots/Code snippets

Mentioned above

Operating System

macOS Sonoma

Python Version

3.12

crewAI Version

0.28.8

crewAI Tools Version

0.16

Virtual Environment

Conda

Evidence