Python libraries: requests and multithreading.dummy

Today was a two-fer: I found two interesting techniques for working with Python. I needed to test a web server, and wanted to send concurrent HTTP get requests. (The problem was that the web server was failing sporadically when hit by a browser, even though individual requests for pages always worked.) I found two libraries that allowed me to write the test code in well under an hour.

  • requests library: I’ve tried urllib2, and it’s a royal pain to use. My general reaction to that library is, “All I want to do is send an HTTP query, and check the response, and I have to write all this code?” Kenneth Reitz has refactored the API so that simple things are simple, and (I suspect, although I didn’t test it), complex things are possible. Check it out at: http://requests.readthedocs.org
  • multithreading.dummy library: Because I wanted to send multiple, overlapping HTTP requests, I figured I’d have to get into threading the Python program. Yuck… I found a great article on Medium https://medium.com/building-things-on-the-internet/40e9b2b36148 that showed how to use the (little known) multithreading.dummy API of Python’s standard multithreading module to trivially (as in, four lines of code) multi-thread the requests.

Leave a Reply