Skip to content
This repository was archived by the owner on Mar 9, 2026. It is now read-only.

feat: add flow control for message publishing#96

Merged
plamut merged 13 commits intogoogleapis:masterfrom
plamut:iss-72
Jun 2, 2020
Merged

feat: add flow control for message publishing#96
plamut merged 13 commits intogoogleapis:masterfrom
plamut:iss-72

Conversation

@plamut
Copy link
Contributor

@plamut plamut commented May 7, 2020

Closes #16.
Closes #72.

This PR adds flow control settings to publisher client, similar to what was added to the Java client recently.

The PR does not interfere with existing publish logic (sequencers, ordering keys, batching...). Instead, it acts like a valve placed in front of the publish pipeline, and taking the configured action if it detects excessive message flow.

Things left to do / discuss

  • A system test for this? Although not sure how feasible would it be, as we would have to publish massive amounts of data on fast connection to have a chance of triggering the flow control error...
  • If the desired behavior is BLOCK, should blocking also support timeouts? However, every publish future gets resolved sooner or later be done (even if due to an error/timeout), and will release the corresponding message from the flow controller then.
  • Unlike Java, the FlowControl settings are placed under PublisherOptions, and not BatchSettings. They fit there much more naturally IMO, but I can change this.
  • What should be the default FlowControl values (the thresholds, specifically)? I set the message count and byte size to 10x the defaults in BatchSettings, would that be fine for most users?
  • Documenting the new feature more extensively than docstring updates in this PR? Can in principle also be done in a separate PR.

PR checklist

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Loading
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla: yes This human has signed the Contributor License Agreement. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PubSub Publisher 504 Deadline Exceeded when sending bulk messages Pubsub: Consider a blocking publisher to prevent OOMing

4 participants