Feature: Support Catalog User Interface by andreashimin · Pull Request #160 · Canner/vulcan-sql

andreashimin · 2023-04-13T10:06:22Z

Description

Support the VulcanSQL Catalog user interface for retrieving data in a more efficient and user-friendly manner.

Sample `vulcan.yaml`

Please notice auth options should be either basic or password-file is setting up or disabled the auth

name: playground1
description: A starter Vulcan project
version: 0.1.0-alpha.1
template:
  provider: LocalFile
  # Path to .sql files
  folderPath: sqls
  codeLoader: InMemory
artifact:
  provider: LocalFile
  serializer: JSON
  # Path to build result
  filePath: result.json
schema-parser:
  reader: LocalFile
  # Path to .yaml files
  folderPath: sqls
document-generator:
  specs:
    - oas3
types:
  - RESTFUL
extensions:
  duckdb: '@vulcan-sql/extension-driver-duckdb'
profiles:
  - profile.yaml
rate-limit:
  options:
    interval:
      min: 1
    max: 10000
enforce-https:
  enabled: false
auth-source:
  options:
    key: token
auth:
  enabled: true
  options:
    basic:
      htpasswd-file:
        path: ./passwd.txt
        users:
          - name: jax
            attr:
              department: engineer
          - name: shimin
            attr:
              department: engineer
    password-file:
      path: ./bcrypt-passwd.txt
      users:
        - name: jax
          attr:
            department: engineer
        - name: shimin
          attr:
            department: engineer
response-format:
  enabled: true
  options:
    default: json
    formats:
      - json
      - csv

Catalog Server

Run Develop Command

yarn nx serve catalog-server

open http://localhost:4200

Default catalog option in yaml

catalog:
   hostname: localhost
   port: 4200

UI

Catalog list

Catalog detail & preview dataset result
<img width="1216" alt="image" src="proxy.php?url=https://user-images.githubusercontent.com/9657305/231646109-102dc036-b46e-425c-982c-afc84e530d61.png->
Connect with third-party services tutorial

VulcanSQL Serve

Run Develop Command

In labs/playground1, run

make start
// or
vulcan start // if already build

open http://localhost:3000

Add /auth/available-types for retrieving auth type by API
Add catalog-routers for providing APIs to Catalog Server.
- /catalog/schemas
- /catalog/schemas/:sourceName
  The schema APIs above will additionally provide url: string, apiDocUrl: string, shareKey: string
- url: host + urlPath in VulcanSQL
- apiDocUrl: the API document url in VulcanSQL
- shareKey: the query string to pass the auth middleware
exclude API docs path in auth middleware

VulcanSQL CLI

After discussion, the package.json will need to be updated after catalog publishing
You will need to do the actions below if you want to test in local

In labs/playground1

Please add the line in package.json

{
   "dependencies": {
       ...
       "@vulcan-sql/catalog-server": "../../dist/packages/catalog-server"
   }
}

Do yarn to install the dependencies
Do make to replace the packages to your local developed packages
You can use vulcan catalog command line in another terminal after running the VulcanSQL server with vulcan start.

VulcanSQL Package

add -t --target for specific which target need to build and the default is vulcan-server

For Catalog Server, you can add the command below

vulcan package --output node --target catalog-server

if user want to adjust the package folder path, can setting it like below

node-packager:
   vulcan-server:
       folderPath: dist
   catalog-server:
       folderPath: dist
docker-packager:
   vulcan-server:
       folderPath: dist
   catalog-server:
       folderPath: dist

Additional Context

vercel · 2023-04-13T10:06:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
vulcan-sql-document	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 17, 2023 7:23am

codecov-commenter · 2023-04-20T02:25:40Z

Codecov Report

Patch coverage: 55.60% and project coverage change: -1.56 ⚠️

Comparison is base (f286a28) 93.01% compared to head (eccb2fe) 91.46%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #160      +/-   ##
===========================================
- Coverage    93.01%   91.46%   -1.56%     
===========================================
  Files          302      313      +11     
  Lines         4755     4943     +188     
  Branches       636      656      +20     
===========================================
+ Hits          4423     4521      +98     
- Misses         214      302      +88     
- Partials       118      120       +2

Flag	Coverage Δ
build	`90.40% <48.48%> (-4.24%)`	⬇️
catalog-server	`100.00% <100.00%> (?)`
cli	`84.92% <50.00%> (-4.47%)`	⬇️
core	`94.36% <ø> (ø)`
extension-dbt	`97.43% <ø> (ø)`
extension-debug-tools	`98.11% <ø> (ø)`
extension-driver-bq	`85.21% <ø> (ø)`
extension-driver-duckdb	`96.46% <ø> (ø)`
extension-driver-pg	`96.11% <ø> (ø)`
extension-driver-snowflake	`96.22% <ø> (ø)`
integration-testing	`90.27% <ø> (ø)`
serve	`86.94% <60.39%> (-2.64%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...generator/spec-generator/oas3/oas3SpecGenerator.ts	`97.46% <ø> (ø)`
packages/serve/src/containers/types.ts	`100.00% <ø> (ø)`
...kages/serve/src/lib/auth/httpBasicAuthenticator.ts	`87.23% <ø> (ø)`
...es/serve/src/lib/auth/passwordFileAuthenticator.ts	`81.81% <ø> (ø)`
...ges/serve/src/lib/auth/simpleTokenAuthenticator.ts	`100.00% <ø> (ø)`
...ages/serve/src/models/extensions/documentRouter.ts	`88.88% <ø> (ø)`
...ges/serve/src/lib/catalog-router/catalogRouters.ts	`26.19% <26.19%> (ø)`
...b/packager/dockerPackager/dockerCatalogPackager.ts	`29.41% <29.41%> (ø)`
...c/lib/packager/nodePackager/nodeCatalogPackager.ts	`31.25% <31.25%> (ø)`
...ve/src/lib/middleware/auth/authRouterMiddleware.ts	`91.48% <33.33%> (-8.52%)`	⬇️
... and 24 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

kokokuo

Besides some suggestions and question, serve,cli and labs package is LGTM 👍

The catalog-server and catalog-server-e2e need @fredalai, @JSYOU to assist, thanks!

kokokuo · 2023-04-25T05:52:43Z

packages/serve/src/lib/catalog-router/catalogRouters.ts

+
+  private getAPIDocUrl(schema: APISchema) {
+    const splitUrl = schema.urlPath.split('/');
+    const restructureUrl = splitUrl


Suggest adding a comment with an example to explain the purpose.

kokokuo · 2023-04-25T05:53:42Z

packages/serve/src/lib/catalog-router/catalogRouters.ts

+      .map((brick) =>
+        brick.startsWith(':') ? `{${brick.replace(':', '')}}` : brick
+      )
+      .join('~1');


Not know why need to join '~1', could you explain it?

That actually the redoc generated url, it will convert the / to ~1
and looked like this one /doc#/paths/~1artist~1%7Bid%7D/get
but I had adjusted the redoc url in 1f05032
now it will more readable: doc#operation/get/artist/:id/works

kokokuo · 2023-04-25T06:05:51Z

packages/serve/src/lib/catalog-router/catalogRouters.ts

+    const project = this.getProjectOptions();
+    const redoc = project?.['redoc'];
+    const docPath =
+      redoc?.url?.replace(/\/+$/, '').replace(/^\/+/, '') || 'doc';


Why should need to add + in the first replace and send replace method? seems you would like to remove the first and last / ?

Btw, the same part also appeared in the AuthCredentialsMiddleware and AuthSourceNormalizerMiddleware, but you not add +

Thanks for finding the difference, I just copied the original code from document router,
It will move to a function and provide the auth refactor function you mention below to use.

kokokuo · 2023-04-25T06:08:48Z

packages/serve/src/lib/middleware/auth/authCredentialsMiddleware.ts

+    // The endpoint not need contains auth credentials
+    const docPrefix =
+      this.projectOptions?.['redoc']?.url
+        .replace(/\/$/, '')
+        .replace(/^\//, '') || 'doc';
+    const pathsWithoutAuth = [
+      '/auth/token',
+      '/auth/available-types',
+      `/${docPrefix}`,
+      `/${docPrefix}/spec`,
+      `/${docPrefix}/redoc`,
+    ];
+    if (pathsWithoutAuth.includes(context.path)) return next();


Seems the AuthSourceNormalizerMiddleware also has the same code, maybe we could refactor the part to a function and put it in utils.ts under the auth folder, how do you think?

Agree, the code had been refactored in 1f05032

kokokuo · 2023-04-25T06:15:59Z

packages/cli/src/commands/catalog.ts

+};
+
+const serveCatalog = async (options: CatalogCommandOptions) => {
+  const next = await import(localModulePath('next'));


Just a question, where is next module from ? node modules ?

Yes, it will retrieve next module from the same local root with catalog-server.

packages/catalog-server/pages/login/index.tsx

packages/catalog-server/components/Layout.tsx

packages/catalog-server/pages/catalog/[slug].tsx

packages/catalog-server/pages/index.tsx

packages/catalog-server/components/catalogDetail/QueryResult.tsx

packages/catalog-server/components/catalogDetail/utils.ts

packages/catalog-server/components/catalogDetail/QueryResult.tsx

packages/catalog-server/components/Layout.tsx

packages/catalog-server/components/catalog/EndpointList.tsx

fredalai

For catalog-server-e2e part, we might be able to remove this part (https://nx.dev/packages/workspace/generators/remove#@nx/workspace:remove) first since we haven't added tests yet, what do you think?

andreashimin · 2023-05-11T07:25:39Z

Hi @fredalai
I have fixed all the suggestions above :D
Please review again when you have free time, thanks~

fredalai

LGTM, except for other comments.

Note: Update your branch based on the latest develop branch finally.

packages/catalog-server/pages/login/index.tsx

packages/catalog-server/components/catalog/index.tsx

kokokuo

Thanks for fixing the last comments!

Beside some suggestions that need to adjust, others LGTM 👍

packages/serve/src/models/extensions/catalogRouter.ts

packages/serve/src/lib/middleware/catalogRouterMiddleware.ts

packages/serve/src/lib/catalog-router/catalogRouters.ts

2. Set graphql package (@apollo/client , graphql-codegen) 3. Add catalog page

…er ui

…e, adjust antd version conflict

…talog ux in config missing

…blic paths to a function

…tside a module

…umentRouter

kokokuo

Thanks for fixing the suggestion, LGTM 👍

Let's please fix the CI test and it could prepare to merge :)

andreashimin requested review from JSYOU, fredalai and kokokuo April 13, 2023 10:06

andreashimin requested review from onlyjackfrost and wwwy3y3 April 13, 2023 10:20

vercel bot deployed to Preview April 20, 2023 02:14 View deployment

kokokuo reviewed Apr 25, 2023

View reviewed changes

vercel bot deployed to Preview April 26, 2023 01:53 View deployment

andreashimin force-pushed the feature/catalog-ui branch from 103dfc7 to 8211ed3 Compare April 26, 2023 02:06

vercel bot deployed to Preview April 26, 2023 02:06 View deployment

vercel bot deployed to Preview May 2, 2023 01:41 View deployment

andreashimin requested a review from kokokuo May 2, 2023 02:02

vercel bot deployed to Preview May 3, 2023 02:50 View deployment

andreashimin force-pushed the feature/catalog-ui branch from f5a250e to 4c3dcd6 Compare May 3, 2023 03:06

vercel bot deployed to Preview May 3, 2023 03:07 View deployment

vercel bot deployed to Preview May 4, 2023 03:00 View deployment

fredalai reviewed May 9, 2023

View reviewed changes

vercel bot deployed to Preview May 10, 2023 10:46 View deployment

vercel bot deployed to Preview May 11, 2023 06:57 View deployment

andreashimin requested a review from fredalai May 11, 2023 07:25

fredalai approved these changes May 15, 2023

View reviewed changes

packages/catalog-server/pages/login/index.tsx Outdated Show resolved Hide resolved

packages/catalog-server/components/catalog/index.tsx Outdated Show resolved Hide resolved

fredalai mentioned this pull request May 15, 2023

Feature: Refine documentation and optimize the usage of configs #165

Merged

kokokuo reviewed May 16, 2023

View reviewed changes

JSYOU and others added 3 commits May 17, 2023 15:06

Add nextjs

2c3657e

feat(catalog): initialize apis with testing data

8c6f65b

1. Set style package (antd, styled-components, styled-system)

d0a1ab0

2. Set graphql package (@apollo/client , graphql-codegen) 3. Add catalog page

andreashimin added 22 commits May 17, 2023 15:13

feat(catalog): restructure vulcan catalog components for sharing cann…

e4fa7b2

…er ui

remove(catalog): styled-system package

6345c59

fix(catalog): lint & remove unused code

a734ede

fix(catalog): change tsconfig alias & setup in tsconfig.base.json

6edb907

feat(catalog): update catalog server api & refine ui

f91e8ee

feat(serve): add catalog router to vulcan-sql

54caf2c

feat(catalog): add tutorial modal for connect from third party servic…

42062b4

…e, adjust antd version conflict

feat(serve): add shareKey to catalog router

bfc2579

feat(serve): change api doc to public route

3501922

feat(catalog): add custom download link, adjust authType check

6ab42c2

fix(serve): spec testing & clean git cached of .tokens.json

5f7a8c2

feat(cli): add catalog command line

17e68e6

feat(catalog, serve): change enpoint name to api url path, enhance ca…

c93a513

…talog ux in config missing

feat(package): add catalog output packager for production build

835ca3f

fix(serve): adjust document router to more readable, refactor auth pu…

9e48abe

…blic paths to a function

fix(build): packager adjust for vulcan.yaml format

2d08229

fix(catalog): remove components dependencies for sharing

7b698c8

fix(catalog): adjust wording & remove redundancy

bf0ce40

remove: catalog-server-e2e

392ee55

fix(catalog): .babelrc will cause jest cannot use import statement ou…

6a99a2e

…tside a module

fix(catalog): remove unused style-components

1b1adb0

fix(serve): remove ambiguous extension setting & add enforceId to doc…

eccb2fe

…umentRouter

andreashimin force-pushed the feature/catalog-ui branch from 770f93c to eccb2fe Compare May 17, 2023 07:23

vercel bot deployed to Preview May 17, 2023 07:23 View deployment

kokokuo approved these changes May 17, 2023

View reviewed changes

andreashimin merged commit f5ee500 into develop May 18, 2023

andreashimin deleted the feature/catalog-ui branch May 18, 2023 06:41

kokokuo mentioned this pull request Jun 12, 2023

Feature: Canner PAT authenticator #181

Merged

Conversation

andreashimin commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Sample vulcan.yaml

Catalog Server

Run Develop Command

UI

VulcanSQL Serve

Run Develop Command

VulcanSQL CLI

VulcanSQL Package

Additional Context

Uh oh!

vercel bot commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Apr 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kokokuo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fredalai left a comment

Choose a reason for hiding this comment

Uh oh!

andreashimin commented May 11, 2023

Uh oh!

fredalai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kokokuo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kokokuo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

andreashimin commented Apr 13, 2023 •

edited

Loading

Sample `vulcan.yaml`

vercel bot commented Apr 13, 2023 •

edited

Loading

codecov-commenter commented Apr 20, 2023 •

edited

Loading

kokokuo left a comment •

edited

Loading