mkflow

• 

  Mkflow

Inspiration

First, this project was inspired by the cloud building stack using serverless/microservice technology which is not available in AWS CodePipeline or Jenkins etc. There was a requirement for CI/CD pipeline with dynamic compute power and this project is exactly what it is for.

Regarding the quarkus, I was mainly got amazed with its whole compatibility with GraalVM stack for Native development. I had already deployed a few small application using Quarkus (private) but this will be my first public project using quarkus. It is simple and easy to develop with it's hot reload feature which is one of the best features I loved. Kudos to the Quarkus.

What it does

This application can dynamically provision a compute power using Cloud provider's like Amazon, Azure(TBD), Google(TBD) at a minimal price (Spot Instance) and run the commands like building and deploying or anything for a limited time and gets auto destroyed after completion.

This can be used for running small batch task eg. If you need one large capacity to compute power for doing a machine learning. Or it can be used for doing our regular code building and deployments and running at its minimal cost.

How I built it

The build stack is quiet simple. I used ssh connection to connect to the provisioned compute machine whereas for the provisioning part I am using Cloud provider's SDK. Every request for running machine is done asynchronously using REST API (Resteasy) and we can see the logs of every task using either CloudWatch Logs or File-based logging.

Challenges I ran into

Mostly the challenges are with the native build. It was tough to get hands-on it at first and do a lot of tests so that it can run in Serverless or Docker container correctly. Mostly it is with the dependencies and libraries that are not fully compatible with Graalvm AOT compilation. There are still some issues so I had to remove and modify the codes differently. The time it takes to compile native is very slow. Overall the project is now fully compatible in native build and runs smoothly.

Accomplishments that I'm proud of

The runtime of the application is just so fast and can be used for serverless application without any hassle. Moreover, I am glad that it runs successfully in AWS Lambda serverless platform. I loved the fact that now I can run CI/CD pipeline with low cost, as well as I can choose now the machine where I can do build and test for faster build time instead of using AWS Codepipeline or Circle CI with everything limited and costly.

What I learned

Learned more about GraalVM and Native deployment and whole architecture of Quarkus.

What's next for mkflow

Need to work on the proper implementation of other cloud providers like Azure, Alibaba, Google etc. Not only that we need the slack integration and proper way to manage the credentials of Github, Gitlab or other Source code repository so that Webhook does not have to worry on sending the critical information in payload.

Built With

• amazon-web-services
• java
• quarkus