tornado.gen — Generator-based coroutines

tornado.gen implements generator-based coroutines.

Note

The “decorator and generator” approach in this module is a precursor to native coroutines (using async def and await) which were introduced in Python 3.5. Applications that do not require compatibility with older versions of Python should use native coroutines instead. Some parts of this module are still useful with native coroutines, notably multi, sleep, WaitIterator, and with_timeout. Some of these functions have counterparts in the asyncio module which may be used as well, although the two may not necessarily be 100% compatible.

Coroutines provide an easier way to work in an asynchronous environment than chaining callbacks. Code using coroutines is technically asynchronous, but it is written as a single generator instead of a collection of separate functions.

For example, here’s a coroutine-based handler:

class GenAsyncHandler(RequestHandler):
    @gen.coroutine
    def get(self):
        http_client = AsyncHTTPClient()
        response = yield http_client.fetch("http://example.com")
        do_something_with_response(response)
        self.render("template.html")

Asynchronous functions in Tornado return an Awaitable or Future; yielding this object returns its result.

You can also yield a list or dict of other yieldable objects, which will be started at the same time and run in parallel; a list or dict of results will be returned when they are all finished:

@gen.coroutine
def get(self):
    http_client = AsyncHTTPClient()
    response1, response2 = yield [http_client.fetch(url1),
                                  http_client.fetch(url2)]
    response_dict = yield dict(response3=http_client.fetch(url3),
                               response4=http_client.fetch(url4))
    response3 = response_dict['response3']
    response4 = response_dict['response4']

If tornado.platform.twisted is imported, it is also possible to yield Twisted’s Deferred objects. See the convert_yielded function to extend this mechanism.

Changed in version 3.2: Dict support added.

Changed in version 4.1: Support added for yielding asyncio Futures and Twisted Deferreds via singledispatch.

Decorators

tornado.gen.coroutine(func: Callable[[...], Generator[Any, Any, _T]]) → Callable[[...], Future[_T]]

tornado.gen.coroutine(func: Callable[[...], _T]) → Callable[[...], Future[_T]]

Decorator for asynchronous generators.

For compatibility with older versions of Python, coroutines may also “return” by raising the special exception Return(value).

Functions with this decorator return a Future.

Warning

When exceptions occur inside a coroutine, the exception information will be stored in the Future object. You must examine the result of the Future object, or the exception may go unnoticed by your code. This means yielding the function if called from another coroutine, using something like IOLoop.run_sync for top-level calls, or passing the Future to IOLoop.add_future.

Changed in version 6.0: The callback argument was removed. Use the returned awaitable object instead.

exception tornado.gen.Return(value: Optional[Any] = None)

Special exception to return a value from a coroutine.

If this exception is raised, its value argument is used as the result of the coroutine:

@gen.coroutine
def fetch_json(url):
    response = yield AsyncHTTPClient().fetch(url)
    raise gen.Return(json_decode(response.body))

In Python 3.3, this exception is no longer necessary: the return statement can be used directly to return a value (previously yield and return with a value could not be combined in the same function).

By analogy with the return statement, the value argument is optional, but it is never necessary to raise gen.Return(). The return statement can be used with no arguments instead.

Utility functions

tornado.gen.with_timeout(timeout: Union[float, datetime.timedelta], future: Yieldable, quiet_exceptions: Union[Type[Exception], Tuple[Type[Exception], ...]] = ())

Wraps a Future (or other yieldable object) in a timeout.

Raises tornado.util.TimeoutError if the input future does not complete before timeout, which may be specified in any form allowed by IOLoop.add_timeout (i.e. a datetime.timedelta or an absolute time relative to IOLoop.time)

If the wrapped Future fails after it has timed out, the exception will be logged unless it is either of a type contained in quiet_exceptions (which may be an exception type or a sequence of types), or an asyncio.CancelledError.

The wrapped Future is not canceled when the timeout expires, permitting it to be reused. asyncio.wait_for is similar to this function but it does cancel the wrapped Future on timeout.

New in version 4.0.

Changed in version 4.1: Added the quiet_exceptions argument and the logging of unhandled exceptions.

Changed in version 4.4: Added support for yieldable objects other than Future.

Changed in version 6.0.3: asyncio.CancelledError is now always considered “quiet”.

Changed in version 6.2: tornado.util.TimeoutError is now an alias to asyncio.TimeoutError.

tornado.gen.sleep(duration: float) → Future[None]

Return a Future that resolves after the given number of seconds.

When used with yield in a coroutine, this is a non-blocking analogue to time.sleep (which should not be used in coroutines because it is blocking):

yield gen.sleep(0.5)

Note that calling this function on its own does nothing; you must wait on the Future it returns (usually by yielding it).

New in version 4.1.

class tornado.gen.WaitIterator(*args: Future, **kwargs: Future)

Provides an iterator to yield the results of awaitables as they finish.

Yielding a set of awaitables like this:

results = yield [awaitable1, awaitable2]

pauses the coroutine until both awaitable1 and awaitable2 return, and then restarts the coroutine with the results of both awaitables. If either awaitable raises an exception, the expression will raise that exception and all the results will be lost.

If you need to get the result of each awaitable as soon as possible, or if you need the result of some awaitables even if others produce errors, you can use WaitIterator:

wait_iterator = gen.WaitIterator(awaitable1, awaitable2)
while not wait_iterator.done():
    try:
        result = yield wait_iterator.next()
    except Exception as e:
        print("Error {} from {}".format(e, wait_iterator.current_future))
    else:
        print("Result {} received from {} at {}".format(
            result, wait_iterator.current_future,
            wait_iterator.current_index))

Because results are returned as soon as they are available the output from the iterator will not be in the same order as the input arguments. If you need to know which future produced the current result, you can use the attributes WaitIterator.current_future, or WaitIterator.current_index to get the index of the awaitable from the input list. (if keyword arguments were used in the construction of the WaitIterator, current_index will use the corresponding keyword).

On Python 3.5, WaitIterator implements the async iterator protocol, so it can be used with the async for statement (note that in this version the entire iteration is aborted if any value raises an exception, while the previous example can continue past individual errors):

async for result in gen.WaitIterator(future1, future2):
    print("Result {} received from {} at {}".format(
        result, wait_iterator.current_future,
        wait_iterator.current_index))

New in version 4.1.

Changed in version 4.3: Added async for support in Python 3.5.

done() → bool

Returns True if this iterator has no more results.

next() → Future

Returns a Future that will yield the next available result.

Note that this Future will not be the same object as any of the inputs.

tornado.gen.multi(Union[List[Yieldable], Dict[Any, Yieldable]], quiet_exceptions: Union[Type[Exception], Tuple[Type[Exception], ...]] = ())

Runs multiple asynchronous operations in parallel.

children may either be a list or a dict whose values are yieldable objects. multi() returns a new yieldable object that resolves to a parallel structure containing their results. If children is a list, the result is a list of results in the same order; if it is a dict, the result is a dict with the same keys.

That is, results = yield multi(list_of_futures) is equivalent to:

results = []
for future in list_of_futures:
    results.append(yield future)

If any children raise exceptions, multi() will raise the first one. All others will be logged, unless they are of types contained in the quiet_exceptions argument.

In a yield-based coroutine, it is not normally necessary to call this function directly, since the coroutine runner will do it automatically when a list or dict is yielded. However, it is necessary in await-based coroutines, or to pass the quiet_exceptions argument.

This function is available under the names multi() and Multi() for historical reasons.

Cancelling a Future returned by multi() does not cancel its children. asyncio.gather is similar to multi(), but it does cancel its children.

Changed in version 4.2: If multiple yieldables fail, any exceptions after the first (which is raised) will be logged. Added the quiet_exceptions argument to suppress this logging for selected exception types.

Changed in version 4.3: Replaced the class Multi and the function multi_future with a unified function multi. Added support for yieldables other than YieldPoint and Future.

tornado.gen.multi_future(Union[List[Yieldable], Dict[Any, Yieldable]], quiet_exceptions: Union[Type[Exception], Tuple[Type[Exception], ...]] = ())

Wait for multiple asynchronous futures in parallel.

Since Tornado 6.0, this function is exactly the same as multi.

New in version 4.0.

Changed in version 4.2: If multiple Futures fail, any exceptions after the first (which is raised) will be logged. Added the quiet_exceptions argument to suppress this logging for selected exception types.

Deprecated since version 4.3: Use multi instead.

tornado.gen.convert_yielded(yielded: Union[None, Awaitable, List[Awaitable], Dict[Any, Awaitable], Future]) → Future

Convert a yielded object into a Future.

The default implementation accepts lists, dictionaries, and Futures. This has the side effect of starting any coroutines that did not start themselves, similar to asyncio.ensure_future.

If the singledispatch library is available, this function may be extended to support additional types. For example:

@convert_yielded.register(asyncio.Future)
def _(asyncio_future):
    return tornado.platform.asyncio.to_tornado_future(asyncio_future)

New in version 4.1.

tornado.gen.maybe_future(x: Any) → Future

Converts x into a Future.

If x is already a Future, it is simply returned; otherwise it is wrapped in a new Future. This is suitable for use as result = yield gen.maybe_future(f()) when you don’t know whether f() returns a Future or not.

Deprecated since version 4.3: This function only handles Futures, not other yieldable objects. Instead of maybe_future, check for the non-future result types you expect (often just None), and yield anything unknown.

tornado.gen.is_coroutine_function(func: Any) → bool

Return whether func is a coroutine function, i.e. a function wrapped with coroutine.

New in version 4.5.

tornado.gen.moment

A special object which may be yielded to allow the IOLoop to run for one iteration.

This is not needed in normal use but it can be helpful in long-running coroutines that are likely to yield Futures that are ready instantly.

Usage: yield gen.moment

In native coroutines, the equivalent of yield gen.moment is await asyncio.sleep(0).

New in version 4.0.

Deprecated since version 4.5: yield None (or yield with no argument) is now equivalent to yield gen.moment.