Skip to main content

Using hooks for custom behaviour in requests

Recently I’ve been writing a lot of scripts with python-requests to interact with a new API. It starts off with a simple GET request:

resp = requests.get('http://example.com/api/v1/assets', params={...})

I want to make sure that the request succeeded before I carry on, so I throw an exception if I got an error responses:

resp = requests.get('http://example.com/api/v1/assets', params={...})
resp.raise_for_status()

If I get an error, the server response may contain useful debugging information, so let’s log that as well (and actually, logging it might be generally useful):

resp = requests.get('http://example.com/api/v1/assets', params={...})

try:
    resp.raise_for_status()
except requests.HTTPError:
    logger.error('Received error %s', resp.text)
    raise
else:
    logger.debug('Received response %s', resp.text)

And depending on the API, I may want even more checks or logging. For example, APIs that always return an HTTP 200 OK, but embedded the real response code in a JSON response. Or maybe I want to log the URL I requested.

If I’m making lots of calls to the same API, repeating this code gets quite tedious. Previously I would have wrapped requests.get in a helper function, but that relies on me remembering to use the wrapper.

It turns out there’s a better way — today I learnt that requests has a hook mechanism that allows you to provide functions that are called after every response. In this post, I’ll show you some simple examples of hooks that I’m already using to clean up my code.

Defining a hook

A hook function takes a Response object, and some number of args and kwargs. For example, if we wanted a hook function that called raise_for_status on every response, this is what we’d write:

def check_for_errors(resp, *args, **kwargs):
    resp.raise_for_status()

If you want requests to call this function on a response, you put it in a dictionary {'response': check_for_errors}. Then you pass this dictionary in the hooks parameter to a requests method:

requests.get(
    'http://example.com/api/v1/assets',
    hooks={'response': check_for_errors}
)

If you want to call multiple hooks, you can also provide a list of functions. For example:

def print_resp_url(resp, *args, **kwargs):
    print(resp.url)

requests.get(
    'http://example.com/api/v1/assets',
    hooks={'response': [print_resp_url, check_for_errors]}
)

Already, this gives us slightly cleaner code — but we still have to remember to use the hooks whenever we make an HTTP request. And what if we think of another hook later, and want to add it to all our existing calls? It turns out there’s an even cleaner way to write this.

Enter the Session API

So far we’ve only used requests’s functional API, but another way to use requests is with Session objects. Using a Session allows you to share cookies, connections and configuration between multiple requests — so it’s already useful if you’re calling the same API repeatedly — and also hooks!

Using the Session API is very similar to the functional API — you create a Session object, then call methods on the object. (In fact, the functional API uses sessions under the hood.) For example:

sess = requests.Session()

sess.get(
    'http://example.com/api/v1/assets',
    hooks={'response': [print_resp_url, check_for_errors]}
)

But Sessions allow us to share configuration between requests. We don’t get any hooks by default:

print(sess.hooks)
# {'response': []}

So instead, we add them after we’ve created the Session object:

sess = requests.Session()
sess.hooks['response'] = [print_resp_url, check_for_errors]

sess.get('http://example.com/api/v1/assets')

and those hooks will now be used for every request made using that session.

Now we can run a consistent set of hooks on every request, but without having repetition throughout our project. And if we update the hooks once, we update the hooks everywhere. Win!

Simple examples of hooks

I’ve already shown you two examples of hooks I’ve written: one to check for errors, another to log the responses.

def check_for_error(resp, *args, **kwargs):
    resp.raise_for_status()


def log_response_text(resp, *args, **kwargs):
    logger.info('Got response %r from %s', resp.text, resp.url)

Because a hook can modify the response object (if it returns a non-None value), you could also use it to edit responses for a more consistent downstream API. Maybe an API that usually returns XML could return JSON instead — but I haven’t experimented with that yet.

Right now, these feel like the most natural use cases for hooks. I wouldn’t want to put business logic in a hook — suddenly hiding the hook implementation makes the code harder to follow — but for simple bookkeeping and error handling, I can see this being really useful.

Now I know that hooks exist, I imagine I’ll find more uses for them. If you want to learn more about hooks, there’s a bit more detail in the requests documentation. I also did a bit of reading of the source code for requests.sessions, but otherwise a bit of experimenting was enough to get me going.