Skip to content

Feature/database as storage#989

Open
ClayLambertExp wants to merge 2 commits intopolynote:masterfrom
QOMPLX-INC:feature/database-as-storage
Open

Feature/database as storage#989
ClayLambertExp wants to merge 2 commits intopolynote:masterfrom
QOMPLX-INC:feature/database-as-storage

Conversation

@ClayLambertExp
Copy link

Implementation on behalf of QOMPLX Inc, for providing database as a storage option for Polynote notebooks. All notebook CRUD operations operate against the database as if it were a file system. Thus all behavior matches that which is provided for File Systems. Whereby notebook data is stored in a database and may be shareable amongst multiple users.

This implementation is based on Postgres as the database store.

@jeremyrsmith
Copy link
Contributor

@ClayLambertExp This is a great option and thanks for implementing it! I think this sort of thing would have to be a plugin, though. Do you mind if we keep this PR open for a bit while we figure out if the plugin infrastructure has enough hooks to do this sort of thing?

*
* Author: Clay Lambert, additions by Ken Hawley
* */
class DatabaseFilesystem(databaseContext: Postgres, maxDepth: Int = 4) extends NotebookFilesystem {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class should be final

@angadsalaria
Copy link

@ClayLambertExp This is a great option and thanks for implementing it! I think this sort of thing would have to be a plugin, though. Do you mind if we keep this PR open for a bit while we figure out if the plugin infrastructure has enough hooks to do this sort of thing?

@jeremyrsmith - Agree on the plugin based modular approach. Our team had same assessment. But wanted to leave to the community on how best to approach it. We'll be leaving the PR as is at this time, and have to move to other endeavors. Happy for you to keep the PR open for various contributors to align & time it with plugin hooks.

* */
class DatabaseFilesystem(databaseContext: Postgres, maxDepth: Int = 4) extends NotebookFilesystem {

var dbContext: DbContext = null

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should be private, also should probably be zio.Ref

* Returns the notebook content from the database wrapped in a Byte oriented stream.
* This makes a synchronous Quill call
*/
private def getNotebookContentStream(path: String): ByteArrayInputStream = {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should suspend the side effects here inside ZIO

/**
* Localized database access functions. Here is where the actual database calls are made.
*/
class Database(dbContext: DbContext) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you suspend all of the database effects here in this class, you can ensure they're all captured in the filesystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants