Making requests
First things first, import the urllib3 module:
>>> import urllib3
instance to make requests. This object handles all of the details of connection pooling and thread safety so that you don’t have to:
>>> http = urllib3.PoolManager()
To make a request use
:
>>> r = http.request('GET', 'http://httpbin.org/robots.txt')
>>> r.data
b'User-agent: *\nDisallow: /deny\n'
object, the Response content section explains how to handle various responses.
You can use
to make requests using any HTTP verb:
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... fields={'hello': 'world'})
The Request data section covers sending other kinds of requests data, including JSON, files, and binary data.
Response content
object provides
, and
attributes:
>>> r = http.request('GET', 'http://httpbin.org/ip')
>>> r.status
200
>>> r.data
b'{\n "origin": "104.232.115.37"\n}\n'
>>> r.headers
HTTPHeaderDict({'Content-Length': '33', ...})
JSON content
JSON content can be loaded by decoding and deserializing the
attribute of the request:
>>> import json
>>> r = http.request('GET', 'http://httpbin.org/ip')
>>> json.loads(r.data.decode('utf-8'))
{'origin': '127.0.0.1'}
Binary content
attribute of the response is always set to a byte string representing the response content:
>>> r = http.request('GET', 'http://httpbin.org/bytes/8')
>>> r.data
b'\xaa\xa5H?\x95\xe9\x9b\x11'
Note
For larger responses, it’s sometimes better to stream the response.
Request data
Headers
You can specify headers as a dictionary in the
argument in
:
>>> r = http.request(
... 'GET',
... 'http://httpbin.org/headers',
... headers={
... 'X-Something': 'value'
... })
>>> json.loads(r.data.decode('utf-8'))['headers']
{'X-Something': 'value', ...}
Query parameters
For
,
, and
requests, you can simply pass the arguments as a dictionary in the
argument to
:
>>> r = http.request(
... 'GET',
... 'http://httpbin.org/get',
... fields={'arg': 'value'})
>>> json.loads(r.data.decode('utf-8'))['args']
{'arg': 'value'}
For
and
requests, you need to manually encode query parameters in the URL:
>>> from urllib.parse import urlencode
>>> encoded_args = urlencode({'arg': 'value'})
>>> url = 'http://httpbin.org/post?' + encoded_args
>>> r = http.request('POST', url)
>>> json.loads(r.data.decode('utf-8'))['args']
{'arg': 'value'}
Form data
For
and
requests, urllib3 will automatically form-encode the dictionary in the
argument provided to
:
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... fields={'field': 'value'})
>>> json.loads(r.data.decode('utf-8'))['form']
{'field': 'value'}
JSON
You can send a JSON request by specifying the encoded data as the
argument and setting the
header when calling
:
>>> import json
>>> data = {'attribute': 'value'}
>>> encoded_data = json.dumps(data).encode('utf-8')
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... body=encoded_data,
... headers={'Content-Type': 'application/json'})
>>> json.loads(r.data.decode('utf-8'))['json']
{'attribute': 'value'}
Files & binary data
For uploading files using
encoding you can use the same approach as Form data and specify the file field as a tuple of
:
>>> with open('example.txt') as fp:
... file_data = fp.read()
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... fields={
... 'filefield': ('example.txt', file_data),
... })
>>> json.loads(r.data.decode('utf-8'))['files']
{'filefield': '...'}
While specifying the filename is not strictly required, it’s recommended in order to match browser behavior. You can also pass a third item in the tuple to specify the file’s MIME type explicitly:
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... fields={
... 'filefield': ('example.txt', file_data, 'text/plain'),
... })
For sending raw binary data simply specify the
argument. It’s also recommended to set the
header:
>>> with open('example.jpg', 'rb') as fp:
... binary_data = fp.read()
>>> r = http.request(
... 'POST',
... 'http://httpbin.org/post',
... body=binary_data,
... headers={'Content-Type': 'image/jpeg'})
>>> json.loads(r.data.decode('utf-8'))['data']
b'...'
Certificate verification
Note
New in version 1.25
HTTPS connections are now verified by default (
<span class="pre">cert_reqs</span> <span class="pre">=</span><span class="pre">'CERT_REQUIRED'</span>).
While you can disable certification verification, it is highly recommend to leave it on.
Unless otherwise specified urllib3 will try to load the default system certificate stores. The most reliable cross-platform method is to use the certifi package which provides Mozilla’s root certificate bundle:
pip install certifi
You can also install certifi along with urllib3 by using the
extra:
pip install urllib3[secure]
Warning
If you’re using Python 2 you may need additional packages. See the section below for more details.
Once you have certificates, you can create a
that verifies certificates when making requests:
>>> import certifi
>>> import urllib3
>>> http = urllib3.PoolManager(
... cert_reqs='CERT_REQUIRED',
... ca_certs=certifi.where())
will automatically handle certificate verification and will raise
if verification fails:
>>> http.request('GET', 'https://google.com')
(No exception)
>>> http.request('GET', 'https://expired.badssl.com')
urllib3.exceptions.SSLError ...
Note
You can use OS-provided certificates if desired. Just specify the full path to the certificate bundle as the
argument instead of
. For example, most Linux systems store the certificates at
. Other operating systems can be difficult.
Certificate verification in Python 2
Older versions of Python 2 are built with an
module that lacks SNI support and can lag behind security updates. For these reasons it’s recommended to usepyOpenSSL.
If you install urllib3 with the
extra, all required packages for certificate verification on Python 2 will be installed:
pip install urllib3[secure]
If you want to install the packages manually, you will need
,
,
, and
.
Note
If you are not using macOS or Windows, note that cryptography requires additional system packages to compile. See building cryptography on Linuxfor the list of packages required.
Once installed, you can tell urllib3 to use pyOpenSSL by using
:
>>> import urllib3.contrib.pyopenssl
>>> urllib3.contrib.pyopenssl.inject_into_urllib3()
that verifies certificates when performing requests:
>>> import certifi
>>> import urllib3
>>> http = urllib3.PoolManager(
... cert_reqs='CERT_REQUIRED',
... ca_certs=certifi.where())
If you do not wish to use pyOpenSSL, you can simply omit the call to
. urllib3 will fall back to the standard-library
module. You may experience several warnings when doing this.
Warning
If you do not use pyOpenSSL, Python must be compiled with ssl support for certificate verification to work. It is uncommon, but it is possible to compile Python without SSL support. See this Stackoverflow thread for more details.
If you are on Google App Engine, you must explicitly enable SSL support in your
:
libraries:
- name: ssl
version: latest
Using timeouts
Timeouts allow you to control how long requests are allowed to run before being aborted. In simple cases, you can specify a timeout as a
to
:
>>> http.request(
... 'GET', 'http://httpbin.org/delay/3', timeout=4.0)
<urllib3.response.HTTPResponse>
>>> http.request(
... 'GET', 'http://httpbin.org/delay/3', timeout=2.5)
MaxRetryError caused by ReadTimeoutError
For more granular control you can use a
instance which lets you specify separate connect and read timeouts:
>>> http.request(
... 'GET',
... 'http://httpbin.org/delay/3',
... timeout=urllib3.Timeout(connect=1.0))
<urllib3.response.HTTPResponse>
>>> http.request(
... 'GET',
... 'http://httpbin.org/delay/3',
... timeout=urllib3.Timeout(connect=1.0, read=2.0))
MaxRetryError caused by ReadTimeoutError
If you want all requests to be subject to the same timeout, you can specify the timeout at the
level:
>>> http = urllib3.PoolManager(timeout=3.0)
>>> http = urllib3.PoolManager(
... timeout=urllib3.Timeout(connect=1.0, read=2.0))
You still override this pool-level timeout by specifying
to
.
Retrying requests
urllib3 can automatically retry idempotent requests. This same mechanism also handles redirects. You can control the retries using the
parameter to
. By default, urllib3 will retry requests 3 times and follow up to 3 redirects.
To change the number of retries just specify an integer:
>>> http.requests('GET', 'http://httpbin.org/ip', retries=10)
To disable all retry and redirect logic specify
:
>>> http.request(
... 'GET', 'http://nxdomain.example.com', retries=False)
NewConnectionError
>>> r = http.request(
... 'GET', 'http://httpbin.org/redirect/1', retries=False)
>>> r.status
302
To disable redirects but keep the retrying logic, specify
:
>>> r = http.request(
... 'GET', 'http://httpbin.org/redirect/1', redirect=False)
>>> r.status
302
For more granular control you can use a
instance. This class allows you far greater control of how requests are retried.
For example, to do a total of 3 retries, but limit to only 2 redirects:
>>> http.request(
... 'GET',
... 'http://httpbin.org/redirect/3',
... retries=urllib3.Retry(3, redirect=2))
MaxRetryError
You can also disable exceptions for too many redirects and just return the
response:
>>> r = http.request(
... 'GET',
... 'http://httpbin.org/redirect/3',
... retries=urllib3.Retry(
... redirect=2, raise_on_redirect=False))
>>> r.status
302
If you want all requests to be subject to the same retry policy, you can specify the retry at the
level:
>>> http = urllib3.PoolManager(retries=False)
>>> http = urllib3.PoolManager(
... retries=urllib3.Retry(5, redirect=2))
You still override this pool-level retry policy by specifying
to
.
Errors & Exceptions
urllib3 wraps lower-level exceptions, for example:
>>> try:
... http.request('GET', 'nx.example.com', retries=False)
>>> except urllib3.exceptions.NewConnectionError:
... print('Connection failed.')
for the full list of all exceptions.
Logging
If you are using the standard library
module urllib3 will emit several logs. In some cases this can be undesirable. You can use the standard logger interface to change the log level for urllib3’s logger:
>>> logging.getLogger("urllib3").setLevel(logging.WARNING)