Skip to content

MINOR: Add async and different sync startup modes in connect service test class#4423

Closed
kkonstantine wants to merge 7 commits intoapache:trunkfrom
kkonstantine:MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class
Closed

MINOR: Add async and different sync startup modes in connect service test class#4423
kkonstantine wants to merge 7 commits intoapache:trunkfrom
kkonstantine:MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class

Conversation

@kkonstantine
Copy link
Contributor

@kkonstantine kkonstantine commented Jan 16, 2018

Allow Connect Service in system tests to start asynchronously.

Specifically, allow for three startup conditions:

  1. No condition - start async and return immediately.
  2. Semi-async - start immediately after plugins have been discovered successfully.
  3. Sync - start returns after the worker has completed startup. This is the current mode, but its condition is improved by checking that the port of Connect's REST interface is open, rather than that a log line has appeared in the logs.

An associated system test run has been started here:
https://jenkins.confluent.io/job/system-test-confluent-platform-branch-builder/586/

@ewencp @rhauch, I'd appreciate your review.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

Copy link
Contributor

@rhauch rhauch Jan 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of this method seems ambiguous to me, even where it's used. Is this following an existing pattern?
Perhaps a comment describing what it does may help. Or, is there an alternative name that might be more clear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add a comment here. But it's equivalent, to the one in zookeeper.py

Copy link
Contributor

@rhauch rhauch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, @kkonstantine. One minor question.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should probably be defined as class constants so users can specify the startup modes without having to use literals. this helps if you have an IDE indexing stuff, and lets you prefix in ways that clarify the meaning, e.g. STARTUP_MODE_LISTEN = 'LISTEN'.

kind of unfortunate we haven't migrated to py3 yet, 3.4 added real enums which would be nicer here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I would have done, but didn't like the naming pattern with all caps. I'll do that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't 8083 basically a constant in this class anyway? what else would we check for?

seems like if we don't want it to be a constant it should be a member variable and not passed in here? other things like _base_url() wouldn't work if you changed it only in the call to this method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more generally, i wouldn't add flexibility until we need it, and I don't see it being used anywhere afaict. one of the nice things about ducktape is that it sets up environments such that your tests don't need flexibility and make things simpler to write.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if these constants aren't variables somewhere, then this should be a docstring instead of a comment above the method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i assume this is used instead of start(self, mode='LISTEN') due to naming conflicts? the name's a bit confusing since in fact both LOAD and LISTEN are synchronous, at least to some degree. i'm not sure just overriding with that extra parameter is even an issue. Would

def start(self, mode='LISTEN'):
    self.startup_mode = mode
    super(ConnectServiceBase, self).start()

not work?

Copy link
Contributor Author

@kkonstantine kkonstantine Jan 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call to the parent method was something that I wasn't sure of ... Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the drawback of renaming here is that the name start_cmd is used with start/start_node pretty standardly across the rest of this code base

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to rename because of a weird error:

cmd = self.start_cmd(node, remote_connector_configs)
[INFO:2018-01-15 17:26:38,306]: TypeError: start_cmd() takes exactly 2 arguments (3 given)

Let's revisit in the future. Right now, renaming was the only way to make this call work generically with both standalone and distributed classes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack. would probably be good to verify the minimal diff to rename back after we're confident of the current patch (and might reveal more info), but this isn't all that big a deal if it changes, just makes it a bit inconsistent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this really what was intended given compatibility? and do none of our tests rely on the existing behavior to wait until that message is logged? i would have expected some that use the REST API (e.g. pretty much anything distributed that submits a connector as one of its first post-start tasks) and would be flaky if we change the default to INSTANT

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, written this way is confusing. I got confused too, before I issue the PR. Indeed, the else might correspond to INSTANT but the thing is that the member variable startup_mode is initialized to LISTEN at the top of the class. So it shouldn't be INSTANT in general and in any of the previous tests, unless explicitly set to that.

But given it's confusing, I'll change it.

@kkonstantine kkonstantine force-pushed the MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class branch from 2369556 to 6937f5f Compare January 17, 2018 01:56
Copy link
Contributor

@ewencp ewencp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a final nit, then I think this LGTM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this docstring isn't applying to what you think it is.

String literals occurring immediately after a simple assignment at the top level of a module, class, or __init__ method are called "attribute docstrings".

This docstring is applying to CONNECT_REST_PORT rather than the subsequent STARTUP_MODE_* constants. This is unfortunate about not having enums in py2. You could either document each separately, or put them under a nested class for namespacing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:) Gotcha

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack. would probably be good to verify the minimal diff to rename back after we're confident of the current patch (and might reveal more info), but this isn't all that big a deal if it changes, just makes it a bit inconsistent.

@kkonstantine
Copy link
Contributor Author

I believe I've addressed the final comments and I've restored start_cmd successfully.

@kkonstantine kkonstantine force-pushed the MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class branch from 6ac49b7 to da728aa Compare January 19, 2018 03:48
@ewencp
Copy link
Contributor

ewencp commented Jan 22, 2018

LGTM, merging to trunk

@ewencp ewencp closed this in 83cc138 Jan 22, 2018
cyrusv pushed a commit to cyrusv/kafka that referenced this pull request Apr 25, 2019
…test class

Allow Connect Service in system tests to start asynchronously.

Specifically, allow for three startup conditions:
1. No condition - start async and return immediately.
2. Semi-async - start immediately after plugins have been discovered successfully.
3. Sync - start returns after the worker has completed startup. This is the current mode, but its condition is improved by checking that the port of Connect's REST interface is open, rather than that a log line has appeared in the logs.

An associated system test run has been started here:
https://jenkins.confluent.io/job/system-test-confluent-platform-branch-builder/586/

ewencp rhauch, I'd appreciate your review.

Author: Konstantine Karantasis <[email protected]>

Reviewers: Randall Hauch <[email protected]>, Ewen Cheslack-Postava <[email protected]>

Closes apache#4423 from kkonstantine/MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class
@cyrusv cyrusv mentioned this pull request Apr 25, 2019
rhauch pushed a commit that referenced this pull request Apr 26, 2019
…tup control (#6638)

This merge consists of two commits previously merged into later branches.
Author: Cyrus Vafadari <[email protected]>
Reviewers: Randall Hauch <[email protected]>

Commit #1:
MINOR: Add async and different sync startup modes in connect service test class

Allow Connect Service in system tests to start asynchronously.

Specifically, allow for three startup conditions:
1. No condition - start async and return immediately.
2. Semi-async - start immediately after plugins have been discovered successfully.
3. Sync - start returns after the worker has completed startup. This is the current mode, but its condition is improved by checking that the port of Connect's REST interface is open, rather than that a log line has appeared in the logs.

Author: Konstantine Karantasis <[email protected]>
Reviewers: Randall Hauch <[email protected]>, Ewen Cheslack-Postava <[email protected]>
Closes #4423 from kkonstantine/MINOR-Add-async-and-different-sync-startup-modes-in-ConnectService-test-class

Commit #2:
MINOR: Modify Connect service's startup timeout to be passed via the init (#5882)

Currently, the startup timeout is hardcoded to be 60 seconds in Connect's test service. Modifying it to be passable via init.

Author: Magesh Nandakumar <[email protected]>
Reviewers: Randall Hauch <[email protected]>, Jason Gustafson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants