Skip to content

Batch Concurrency#358

Merged
deanq merged 7 commits intomainfrom
batch-concurrency
Sep 26, 2024
Merged

Batch Concurrency#358
deanq merged 7 commits intomainfrom
batch-concurrency

Conversation

@deanq
Copy link
Copy Markdown
Member

@deanq deanq commented Sep 25, 2024 •
edited
Loading

  • Restored concurrency_modifier support
  • Concurrent calls now use batched job takes instead of multiple parallel singular job takes
  • Using asyncio.Queue to track jobs queue and processing
  • JobScaler refactored to properly coordinate job takes and processing

@deanq deanq marked this pull request as ready for review September 26, 2024 05:30
@deanq deanq merged commit f1e6450 into main Sep 26, 2024
@deanq deanq deleted the batch-concurrency branch September 26, 2024 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yhlong00 Yhlong00 Yhlong00 approved these changes

@justinmerrell justinmerrell Awaiting requested review from justinmerrell

@ef0xa ef0xa Awaiting requested review from ef0xa

@rutvik-runpod rutvik-runpod Awaiting requested review from rutvik-runpod

@flash-singh0 flash-singh0 Awaiting requested review from flash-singh0

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@deanq @Yhlong00