[dbsp] Make DevTweaks public so it will get documented. by blp · Pull Request #5941 · feldera/feldera

blp · 2026-03-27T22:24:11Z

Describe Manual Test Plan

I ran the unit tests.

mythical-fred

LGTM. Clean extraction. The move to Option<T> fields with explicit accessors is the right approach for a public, documented struct.

mihaibudiu · 2026-03-27T23:18:45Z

crates/feldera-types/src/config/dev_tweaks.rs

+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
+
+/// Optional settings for tweaking Feldera internals.


Do you want to say something about how these are not guaranteed to be a stable API?

mihaibudiu · 2026-03-27T23:19:27Z

crates/pipeline-manager/src/db/types/program.rs

-        multihost: if runtime_config.hosts > 1
-            || runtime_config.dev_tweaks.contains_key("multihost")
-        {
+        multihost: if runtime_config.hosts > 1 {


so this one is gone?

This was an experiment that I used for testing while developing the multihost feature. I forgot about it. It's not needed anymore.

mihaibudiu · 2026-03-27T23:21:53Z

crates/feldera-types/src/config/dev_tweaks.rs

+    /// storage, such as object storage, but it could use excessive amounts of
+    /// memory if the number of keys fetched is very large.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub fetch_join: Option<bool>,


The name does not suggest the same thing that the documentation describes, but it's probably too late to change that.

mihaibudiu · 2026-03-27T23:22:12Z

crates/feldera-types/src/config/dev_tweaks.rs

+    /// storage, such as object storage, but it could use excessive amounts of
+    /// memory if the number of keys fetched is very large.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub fetch_distinct: Option<bool>,


same as before

mihaibudiu · 2026-03-27T23:23:12Z

crates/feldera-types/src/config/dev_tweaks.rs

+    /// each step.
+    ///
+    /// The default value is 10,000 records.
+    // TODO: splitter_chunk_size_bytes, per-operator chunk size.


I don't understand this TODO.

I don't know this either. It is a copy of a @ryzhyk comment, so he might chime in.

The TODO suggests that we support configuring chunk size in bytes vs records, and that we may want to make it configurable per operator. It's not part of the doc comment. I don't think this is something we need to worry about now.

I changed this to:

// TODO: It would be better if the value were denominated in bytes rather // than records, and if it were configurable per-operator.

mihaibudiu · 2026-03-27T23:25:06Z

crates/feldera-types/src/config/dev_tweaks.rs

+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub adaptive_joins: Option<bool>,
+
+    /// The minimum relative improvement threshold for the join balancer.


I think "adaptive" and "balanced" are the same thing, but the names of the fields do not make it clear. Perhaps the comments should.

I think I need @ryzhyk's feedback on that too.

We can add this sentence to the doc comment:

"The join balancer is a component that dynamically chooses an optimal partitioning policy for adaptive join operators."

mihaibudiu · 2026-03-27T23:26:56Z

crates/feldera-types/src/config/dev_tweaks.rs

+        self.adaptive_joins.unwrap_or(false)
+    }
+
+    pub fn balancer_min_relative_improvement_threshold(&self) -> f64 {


why sometimes empty lines and sometimes no?

Mistake. I will fix it.

anandbraman

Thanks for documenting!

Signed-off-by: Ben Pfaff <[email protected]>

Signed-off-by: feldera-bot <[email protected]>

blp requested review from anandbraman and gz March 27, 2026 22:24

blp self-assigned this Mar 27, 2026

blp added documentation Improvements or additions to documentation DBSP core Related to the core DBSP library API Distributed system APIs User-facing For PRs that lead to Feldera-user visible changes user-reported Reported by a user or customer labels Mar 27, 2026

mythical-fred approved these changes Mar 27, 2026

View reviewed changes

mihaibudiu approved these changes Mar 27, 2026

View reviewed changes

gz approved these changes Mar 28, 2026

View reviewed changes

anandbraman approved these changes Mar 30, 2026

View reviewed changes

blp force-pushed the devtweaks branch 2 times, most recently from db740b8 to 62f881d Compare March 31, 2026 16:24

blp enabled auto-merge March 31, 2026 16:24

[dbsp] Make DevTweaks public so it will get documented.

1206364

Signed-off-by: Ben Pfaff <[email protected]>

blp force-pushed the devtweaks branch from 9c54b2f to 1206364 Compare March 31, 2026 16:50

[ci] apply automatic fixes

3c6c10f

Signed-off-by: feldera-bot <[email protected]>

blp added this pull request to the merge queue Mar 31, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 31, 2026

blp added this pull request to the merge queue Mar 31, 2026

Merged via the queue into main with commit 68ee6dd Apr 1, 2026
1 check passed

blp deleted the devtweaks branch April 1, 2026 00:25

Conversation

blp commented Mar 27, 2026

Describe Manual Test Plan

Uh oh!

mythical-fred left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandbraman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants