sprockets.mixins.http¶
HTTP Client Mixin for Tornado RequestHandlers. Automatically retries on errors, sleep when rate limited, and handles content encoding and decoding using MsgPack and JSON.
Installation¶
sprockets.mixins.http is available on the
Python Package Index
and can be installed via pip:
pip install sprockets.mixins.http
If you would like to use tornado.curl_httpclient.CurlAsyncHTTPClient, you can install pycurl with:
pip install sprockets.mixins.http[curl]
Documentation¶
Requirements¶
- ietfparse >=1.5.1
- tornado >=5
- sprockets.mixins.mediatype[msgpack] >=3
Example¶
This examples demonstrates the most basic usage of sprockets.mixins.http
from tornado import ioloop, web
from sprockets.mixins import http
class RequestHandler(http.HTTPClientMixin, web.RequestHandler):
async def get(self, *args, **kwargs):
response = await self.http_fetch('https://api.github.com')
if not response.ok:
self.set_status(response.code)
self.write(response.body)
if __name__ == "__main__":
app = web.Application([(r'/', RequestHandler)])
app.listen(8000)
ioloop.IOLoop.current().start()
As with Tornado, to use the curl client which has numerous benefits:
from tornado import httpclient, ioloop, web
from sprockets.mixins import http
httpclient.AsyncHTTPClient.configure(
'tornado.curl_httpclient.CurlAsyncHTTPClient')
class RequestHandler(http.HTTPClientMixin, web.RequestHandler):
async def get(self, *args, **kwargs):
response = await self.http_fetch('https://api.github.com')
if not response.ok:
self.set_status(response.code)
self.write(response.body)
if __name__ == "__main__":
app = web.Application([(r'/', RequestHandler)])
app.listen(8000)
ioloop.IOLoop.current().start()
Error Response Body¶
For errors, i.e. a response with HTTP status code in the 400 range…
The HTTPResponse object’s body is reduced down to just the error message. That is this mixin’s default behavior.
For a JSON response body with Problem Details (RFC 7807), you may want more than just the error message. To gain access to the complete, deserialized response body; a class that uses this mixin can set:
self.simplify_error_response = False
Environment Variables¶
| HTTP_MAX_CLIENTS | An optional setting that specifies the maximum number of simultaneous asynchronous HTTP requests. If not specified, the default Tornado value of 10 will be used. |
License¶
sprockets.mixins.http is released under the 3-Clause BSD license.
Issues¶
Please report any issues to the Github project at https://github.com/sprockets/sprockets.mixins.http/issues
API¶
-
class
sprockets.mixins.http.HTTPClientMixin(*args, **kwargs)¶ Mixin for making http requests using the asynchronous
http_fetch()method.-
_http_req_apply_default_headers(request_headers, content_type, body)¶ Set default values for common HTTP request headers
Parameters: - request_headers (dict) – The HTTP request headers
- content_type (
ietfparse.datastructures.ContentTypeor str) – The mime-type used in the request/response - body (mixed) – The request body
Return type:
-
_http_req_body_serialize(body, content_type)¶ Conditionally serialize the request body value if mime_type is set and it’s serializable.
Parameters: - body (mixed) – The request body
- content_type (str) – The content type for the request body
Raises: ValueError
-
http_fetch(url, method='GET', request_headers=None, body=None, content_type=<ietfparse.datastructures.ContentType application/msgpack, 0 parameters>, follow_redirects=False, max_redirects=None, connect_timeout=None, request_timeout=None, retry_timeout=None, max_http_attempts=None, auth_username=None, auth_password=None, user_agent=None, validate_cert=True, allow_nonstandard_methods=False, dont_retry=None, **kwargs)¶ Perform a HTTP request
Will retry up to max_http_attempts times with an exponentially increasing sleep time starting with retry_timeout seconds. If a
Retry-Headeris included in a response, then it will override the calculated sleep time.Parameters: - url (str) – The URL for the request
- method (str) – The HTTP request method, defaults to
GET - request_headers (dict) – Headers to include in the HTTP request
- body (mixed) – The HTTP request body to send with the request
- content_type (
ContentTypeor str) – The mime type to use for requests & responses. Defaults toapplication/msgpack - follow_redirects (bool) – Follow HTTP redirects when received
- max_redirects (int) – Maximum number of redirects to follow, default is 5
- connect_timeout (float) – Timeout for initial connection in seconds, default 20 seconds
- request_timeout (float) – Timeout for entire request in seconds, default 20 seconds
- retry_timeout (float) – Time to sleep between retries, default 3 seconds
- max_http_attempts (int) – Maximum number of times to retry a request, default is 3 attempts
- auth_username (str) – Username for HTTP authentication
- auth_password (str) – Password for HTTP authentication
- user_agent (str) – The str used for the
User-Agentheader, default used if unspecified. - validate_cert (bool) – For HTTPS requests, validate the server’s certificate? Default is True
- allow_nonstandard_methods (bool) – Allow methods that don’t adhere to the HTTP spec.
- dont_retry (set) – A list of status codes that will not be retried if an error is returned. Default: set({})
- kwargs – additional keyword parameters are passed to
tornado.httpclient.AsyncHTTPClient.fetch()
Return type: Raises: RuntimeErrorif theraise_errorkeyword argument is specified
-
_http_req_user_agent()¶ Return the User-Agent value to specify in HTTP requests, defaulting to
service/versionif configured in the application settings, or if used in a consumer, it will attempt to obtain a user-agent from the consumer’s process. If it can not auto-set the User-Agent, it defaults tosprockets.mixins.http/[VERSION].Return type: str
-
_http_req_modify_for_retry(response: sprockets.mixins.http.HTTPResponse, attempt: int, url: str, headers: dict, body)¶ Implement this method to modify the request on each attempt.
Parameters: - response – the current HTTP response which includes both response and exception history
- attempt – current attempt counter
- url – current request URL
- headers – current request headers
- body – serialized request body
The default behavior is to add the
X-Retry-Attemptheader. You will need to implement this for protocols that include a one-time-use value such as the OAuth 1 request nonce.Returns: a tuple containing the URL, headers, and body to use in the next request
-
-
class
sprockets.mixins.http.HTTPResponse(simplify_error_response=True)¶ Encapsulate the response(s) for requests made using the
http_fetch()method.-
attempts¶ Return the number of HTTP attempts made by calculating the number of exceptions and responses the object contains.
Return type: int
-
body¶ Returns the HTTP response body, deserialized if possible.
Return type: mixed
-
duration¶ Return the calculated duration for the total amount of time across all retries.
Return type: float
-
exceptions¶ Return the list of exceptions raised when making the request.
Return type: list(Exception)
-
history¶ Return all of the HTTP responses for the request.
Return type: list(tornado.httpclient.HTTPResponse)
-
links¶ Return the parsed link header if it was set, returning a list of the links as a dict.
Return type: list(dict()) or None
-
ok¶ Returns True if the response status code was between 200 and 399. Returns False if no responses were received or the response status code was >= 400.
:rtype bool
-
raw¶ Return the raw tornado HTTP Response object
Return type: tornado.httpclient.HTTPResponse
-
Version History¶
2.6.0 Mar 07 2022¶
- Add support for
Content-Typesuffixes (ex.a/b+json,x/y+msgpack) - Fix exception when
self.correlation_idisNone
2.5.0 Sep 16 2021¶
- Change
HTTPResponse.linksto return empty list whenLinkheader is not present - Move
X-Retry-Attemptheader insertion into_http_req_modify_for_retry()hook. This is also needed for using the client with OAuth 1 servers.
2.4.1 Nov 30 2020¶
- Make request retry timeout configurable
- Apply retry sleeping to all retried attempts
- Use an exponential backoff if
Retry-Afterheader is absent - Add
retry_timeoutparameter tohttp_fetch()
2.4.0 Nov 3 2020¶
- Fix serialization of empty request bodies.
- Rate limit 503s as well as 423s and 429s.
- Advertise & test support for Python 3.8 and 3.9.
2.3.3 Apr 8 2020¶
- Pass keyword parameters through to the underlying HTTPClient fetch method. This enables niceties like streaming callback support
2.3.0 Dec 9, 2019¶
- Added an option to control response body transformation for errors, i.e. HTTP
status code >= 400. By default, a JSON or otherwise structured response body
will be reduced down to its error
message. That can be overridden by settingsimplify_error_responseto False. - Fix compile-time setting of default argument values in
http_fetch.
2.2.0 Aug 29, 2019¶
- Add handling of
tornado.httpclient.HTTPTimeoutErrorandtornado.httpclient.HTTPStreamClosedErrorexceptions - Fix documentation builds
- Update documentation links to readthedocs.io
2.1.0 May 7, 2019¶
- Cast the
urlparameter ofhttp_fetchto a string. Allows for native use of URL abstractions like yarl.
2.0.0 Apr 1, 2019¶
- Refactor the HTTPResponse to a stand-alone class
- Add
historyattribute of the response with all response objects - Addlinksattribute of the response with the parsed link header if set - Addexceptionsattribute with stack of exceptions returned as responses - Add
dont_retryas argument tohttp_fetchmethod - Change logging level in a few places to a more appropriate level
- Add support for rejected consumers when auto-creating the
User-Agentheader - Add the netloc of a request to the log entry created when rate limited
- Use RequestHandler.settings instead of RequestHandler.application.settings
when auto-creating the
User-Agentheader for a Tornado request handler - Add test coverage of the Warning response header behavior