How to save and load cookies in Python requests?

by scrapecrow Dec 08, 2022

HTTP cookies in web scraping play a vital role across several factors, including blocking and localization. Hence, pausing and resuming request sessions is often useful. For this, we'll explain how to save and load Python requests cookies.

Save Cookies

Let's start by saving cookies in Python requests. For this, we'll use the requests.cookiejar utility package:

from pathlib import Path
from requests.utils import dict_from_cookiejar
import json
import requests

# create a session to presists cookies
session = requests.session()

# send cookies through a response object
session.get("https://httpbin.dev/cookies/set?key1=value1&key2=value2")

# turn cookiejar into dict
cookies = dict_from_cookiejar(session.cookies)  

# save them to file as JSON
Path("cookies.json").write_text(json.dumps(cookies))

Above, we create a new session object and request a URL to accept few cookie values. Then, we save the retrieved Python request cookies to an object using the dict_from_cookiejar method. Finally, the cookie data is saved to a JSON file:

{"key1": "value1", "key2": "value2"}

Load Cookies

Now that we saved the cookie object using Python cookiejar, let's load it:

from pathlib import Path
from requests.utils import cookiejar_from_dict
import json
import requests

# create a session
session = requests.session()

# load the JSON file
cookies = json.loads(Path("cookies.json").read_text())

# turn the object into a a cookie jar
cookies = cookiejar_from_dict(cookies)

# load cookiejar to current session
session.cookies.update(cookies)

# validate the cookies
print(session.get("https://httpbin.dev/cookies").text)
{"key1": "value1", "key2": "value2"}

Here, we started by loading the JSON file and importing the cookie values using the cookiejar_from_dict. Next, we let the requests set cookie values by loading them into the session, which will automatically set cookie header values.

For further details on utilizing cookies for requests in Python, refer to our dedicated guide.