Telling mechanize how to find local issuer certificates
I was doing some work on my library lookup tool recently, and I ran into an issue with mechanize, the Python library I use to simulate a web browser. I’d upgraded my version of Python and mechanize, and now I wasn’t able to connect to HTTPS sites.
If I tried a simple example:
import mechanize
browser = mechanize.Browser()
browser.set_handle_robots(False)
browser.open("https://www.example.net/").read()
it would fail with an error:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed: unable to get local issuer certificate (_ssl.c:1000)
My first instinct was to check Google and GitHub; I couldn’t find any other instances of people finding and fixing this issue. The most I could find was a quickstart example that starts with an HTTP example, then suggests disabling SSL verification to access HTTPS sites. I found a few instances of people following this suggestion on GitHub – but I wasn’t keen on that. SSL verification exists for a reason; I don’t want to get rid of it!
A bit later I found a page about changing the certificates used by your mechanize browser with browser.set_ca_data(). I knew from my work on HTTP libraries that certifi is a bundle of SSL certificates often used in Python libraries, so I decided to try pointing mechanize at certifi:
import mechanize
import certifi
browser = mechanize.Browser()
browser.set_handle_robots(False)
+browser.set_ca_data(cafile=certifi.where())
browser.open("https://www.example.net/").read()
That seemed to work, and my mechanize browser was once again able to browse the HTTPS web.
If you use popular HTTP libraries like httpx or requests, they install and load SSL certificates from certifi automatically. I don’t know why mechanize doesn’t do the same, but it was just a one-line change to get it working correctly.