This project is focused on MQTT communication, IPFS,MLFLOW,DJANGO BACKED TO STORE ON LOCAL STORAGE and Decentralized FL.
If you use or refer to this project in your work, please cite the following paper:
Shrestha, A. K., Jaberzadeh, A., Khan, F. A., Geng, J., & Shaikh, M. A. (2023). Enhancing Scalability and Reliability in SemiDecentralized Federated Learning With Blockchain: Trust Penalization and Asynchronous Functionality. arXiv. https://arxiv.org/pdf/2310.19287
AWS Config
-
Create an AWS EC2 Instance:
To create an EC2 instance with Ubuntu, machine type t2.2xlarge, and 30 GB SSD storage, follow these steps:
-
Open the AWS Management Console:
- Go to AWS Console.
-
Navigate to EC2:
- In the AWS Console, click on "Services" and then select "EC2" under the Compute section.
-
Launch an Instance:
- Click on the "Instances" link in the left sidebar.
- Click the "Launch Instances" button.
-
Choose an Amazon Machine Image (AMI):
- In the "Choose an Amazon Machine Image (AMI)" step, select "Ubuntu" as the operating system.
-
Choose an Instance Type:
- In the "Choose an Instance Type" step, scroll down and select "t2.2xlarge."
-
Configure Instance:
- In the "Configure Instance" step, leave the default settings or adjust them as needed.
-
Add Storage:
- In the "Add Storage" step, set the size to "20 GiB" and choose "SSD" as the volume type.
-
Add Tags:
- In the "Add Tags" step, add any tags if necessary.
-
Configure Security Group:
- In the "Configure Security Group" step, configure security group rules to allow incoming traffic on the desired ports (e.g., SSH on port 22, HTTP on port 80, etc.).
-
Review and Launch:
- In the "Review" step, review your configuration.
-
Launch:
- Click the "Launch" button.
-
Create a Key Pair:
- In the pop-up window, select "Create a new key pair" from the drop-down.
- Enter a key pair name and click "Download Key Pair."
-
Launch Instances:
- Click the "Launch Instances" button.
-
View Instances:
- Click on "View Instances" to see the status of your launched instance.
-
Connect to the Instance:
- Once the instance is running, select the instance, and click the "Connect" button.
- Follow the instructions to connect using SSH, using the downloaded key pair.
Your EC2 instance with Ubuntu, machine type t2.2xlarge, and 20 GB SSD storage should now be running. Adjust security group rules and other settings as needed for your specific requirements.
-
-
Change Inbound Rules to Open Ports:
In the AWS Console, go to the EC2 Dashboard, and select your instance.
-
Click on the "Security" tab.
-
Under "Security groups," click on the associated security group.
-
In the "Inbound rules" tab, add rules to open ports 5000 and 8000 for mlflow and Django server:
Type Protocol Port Range Source Custom TCP TCP 5000 0.0.0.0/0 Custom TCP TCP 8000 0.0.0.0/0
-
-
Allocate and Associate an Elastic IP (Optional):
If you want a static IP for your instance, allocate and associate an Elastic IP:
- In the AWS Console, go to the EC2 Dashboard.
- Under "Network & Security," click on "Elastic IPs."
- Allocate a new Elastic IP and associate it with your EC2 instance.
To run the project, follow the steps below:
Clone this repository to your local machine.
git clone https://github.com/bayesianinstitute/Bayes_DFL.gitNavigate to the project directory.
cd FL_Mock- Install pip and virtual env
sudo pip install virtualenv- Create a virtual environment
virtualenv myenv- Activate the Virtual Environment:
source myenv/bin/activate- Install Packages and Run Your Project:
pip install -r requirements.txtbash run run_ipfs.shWorkflow Info
The DFL workflow implemented in this script consists of the following steps:
- Participant identification.
- MQTT setup and communication.
- Training machine learning models.
- Aggregation of models (if the participant is the aggregator).
- Model validation and iteration.
The workflow can be customized based on your specific use case and requirements.
The project's core contains the main MqttOPS class that handles the MQTT cluster communication. The core functionalities include:
- Creating and managing MQTT clients.
- Running the logic for the cluster.
Run Django server:
Certainly! Here's the modified set of instructions including migrating, makemigrations, creating a superuser, and running the Django server:
# Create database migrations:
python db/manage.py makemigrations bayes_app
# Migrate the database:
python db/manage.py migrate bayes_app
# Apply migrations:
python db/manage.py migrate
# Create a superuser:
python db/manage.py createsuperuser
# Run Django server:
python db/manage.py runserver 0.0.0.0:8000Make sure to follow these steps after activating the virtual environment. This sequence will migrate the database, create necessary migrations, prompt you to create a superuser, and finally run the Django development server on 0.0.0.0:8000. Adjust the steps according to your project's requirements.
Run mlflow:
cd core/MLOPS/Model/
mlflow ui --host 0.0.0.0 --port 5000To execute the main program, follow these steps:
-
Open your terminal.
-
Navigate to the project directory.
-
Use the following command to run the script:
bash run.sh TRAINING_NAME ROLE
Replace
TRAINING_NAMEwith the desired training name (e.g., "UCLA") andROLEwith either "Admin" or "User."Example:
bash run.sh UCLA User
This command initiates the program with the specified training name and role, allowing you to customize the execution based on your requirements.