@alexey-milovidov Alex! Really appreciated for your great wishes from ClickHouse team!

In terms of general product strategy, direction?

In terms of general product strategy, there are many information not released indeed. From the detailed tech implementation, ClickHouse(CH, for short), of course, is our great teacher. But, furthermore, TensorBase(TB, for short) also wants to systematically correct the problems shown in ClickHouse and from the view of my seven-year experiences of bigdata system development and operation.

These projects look very similar, I'm wondering how do they differ.

Datafuse(DFe, for short) is a sad topic. I am used to invite the team of DFe several months ago. But they've got good memory from local ventures. The invitation was not accepted.

Before this invitation, the project is licensed under a commercially unfriendly license AGPL. Personally, I do not lookup any source of commercially unfriendly licensed project. Before this invitation, I have built one prototype of pure Rust engine based on the Arrow and DataFusion. And it works greatly and become the new initial base of TB as you seen.

After the time of invitation, I evaluated the DFe which has been changed to the APL licence. The conclusion is, basically there is nothing new in DFe except for a good engineering structure (but a nice engineering structure is a low-hanging fruit in my humble opinion):

Toy engine with misleading benchmarks on top of toy dataset
Misleading name: DataFuse v.s. DataFusion?
Its route: Arrow + its own engine. But, its engine, in fact, reinvents the wheel of DataFusion which is now the base of TensorBase.
DFe also repeats the old story in other respects from the history IMHO.

In TensorBase, we have a heart of change. This is the fundamental difference to DFe from TB.

Repeating makes no sense.
DataFusion is already a repeating to existed engine story: Spark, Presto, Impala ... and our ClickHouse (orz, CH is not only an engine). To use raft library is repeating. To have new hash-join operator with a repeating hashmap library is repeating. To clone a CH's MergeTree engine is repeating.

Do we have thought: is raft great for the progress of this era? is hash-join great for the progress of this era? is MergeTree great for the progress of this era?

Do you think: a CH Rust-clone makes sense for the world?

TensorBase's answer is: No.
Help to the community rather than reinvention-the-wheel unless we have new.
TB chooses to help all the existed community, and DFe chooses to repeat an old thing in its new codes in a little misleading naming 😄

Back to the first question, TensorBase, in fact, has pinned a very different way to ClickHouse and of course to CH's clones :

TB will not use LSM or Mergetree. Your mentioned DFe even has no storage now. TB has more efficient Partition-Tree based storage than CH. (soon we demonstrate more.)
TB will not use the raft protocol for distribution. CH and your mentioned DFe used. But we want to show we still do not compromise the consistency and top performance. (soon)
TB will soon show our engine optimization fruit on top of DataFusion, for which we show to challenge more of CH performance in real world dataset.
TB propose a data-control subsystem from its day one to conquer the security and privacy of bigdata.
TB has a revolutionary engine as you seen in the frontier edition. In that engine, we have different scheduler, runtime which may not exist in the world, at least in the open-source world. I hope to bring it back to the community to service for this era like CH done currently.
...

Several local ventures ask me a question: if DFe and similar projects copy your codes, then how about you? I have no idea. Your CH has been copied everywhere but it has Yandex backed. This is not the case for TB.

So, for your first question, my current answer is: we reveal them when they come. It is hoped that, if they copy, the world knows that things are copied from TB.

Finally, I am not sure, if there is an opportunity to work with the Yandex/CH team in the next-generation data warehouse from the new engineering of TB. TB now is a good friend in CH ecosystem. TB is open for any possibility.

How it is different with Datafuse? #207

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions