Blogs Engineering

Letting YDB's Topic Reader Close When the Stream Breaks

YDB's Python topic reader could hang on commit acknowledgements after a stream error. The merged fix stops the flush wait when the first error arrives.

Pythonydb-platform/ydb-python-sdkasynciotopic readershutdown

A reader cannot finish shutting down if it waits for acknowledgements from a stream that has already failed.

Short answer

YDB's Python topic reader could hang in close(flush=True) when a broken stream still had uncommitted offsets pending. The flush waited for commit acknowledgements that could never arrive. The merged PR #920 makes ReaderStream.flush() stop waiting when either the acknowledgements finish or the stream's first error is set. It closes issue #679.

The problem

The original report described a topic reader whose close method froze indefinitely. The reporter used SDK version 3.21.2, Python 3.12, and Ubuntu on x86_64, and supplied a producer-consumer reproduction. The expected behavior was for close to complete in a reasonable time.

Root cause

ReaderStream.flush() collected the partition sessions' acknowledgement futures and awaited asyncio.wait(futures). Once the stream broke, those acknowledgements could remain unresolved. Waiting for all of them prevented close from reaching stream teardown.

The PR description identifies the wait on an already-broken stream as the cause. The change races the acknowledgement wait against self._first_error.

The fix

The merged diff replaces the unconditional acknowledgement wait with:

if not futures:
    return

all_acks_received = asyncio.ensure_future(asyncio.wait(futures))
try:
    await asyncio.wait((all_acks_received, self._first_error), return_when=asyncio.FIRST_COMPLETED)
finally:
    all_acks_received.cancel()

An empty set of acknowledgement futures returns immediately. Otherwise, the first completed condition releases the wait. The finally block cancels the task wrapping the acknowledgement wait.

Two regression tests cover the failure path. test_flush_stops_on_stream_error checks that flush returns after a stream error while the pending acknowledgement remains unresolved, then verifies that close fails the waiter with PublicTopicReaderPartitionExpiredError. test_close_with_flush_on_broken_stream_does_not_hang sets a stream error before calling close(True) and checks the same waiter exception.

By the numbers

  • Repository: ydb-platform/ydb-python-sdk
  • Files changed: 3, including the implementation, tests, and changelog
  • Lines: +36 / -3
  • Commits: 3
  • Opened: October 10, 2026. Merged: November 5, 2026
  • Review: approved by a project member before merge

How the review went

The public coverage report states: "All modified and coverable lines are covered by tests." That is the report's statement, rather than an independent test run for this article.

The automated review also identified a coverage limitation: the regression test does not start a flush task and then inject a later stream error while that task is waiting. The merged tests exercise an already-broken stream. A project member approved the PR on November 5, 2026, and the PR was merged that day.

Takeaways

  • Wait for failure as well as success. A shutdown flush needs an exit when its transport can no longer deliver acknowledgements.
  • Distinguish test scenarios. An already-failed stream and a stream that fails during a wait are separate cases.
  • Keep teardown reachable. Pending waiters can receive a meaningful exception only after the code reaches cleanup.

FAQ

Does this change wait for every acknowledgement after a stream error?

No. It stops the flush wait when self._first_error completes, so close can proceed with teardown. The regression tests expect pending waiters to fail with PublicTopicReaderPartitionExpiredError.

Where can I inspect the change?

Read the pull request, diff and tests, and original issue.

The goal

A failed transport should not leave shutdown waiting forever for a successful response. This fix gives the topic reader's flush an error exit so close can finish cleaning up.

Run the loop

Turn production signals into reviewed fixes.

Start a free trial and see how Prilog maps real incidents to code-level pull requests.