Below is a typical setup for this end-to-end test:

There are three parts in this test setup:

1. Alcor services, mainly running Network Configuration Manager(NCM) and Ignite.

2. Compute nodes, mainly running Alcor-Control-Agent.

3. Arion Control Plane and Data Plane, including Arion Master(and its Hazelcast Database), Arion Agent and Arion DP Controller.

1. Test Controller reads its configuration file and generates metadata, such as how many VPCs, Subnets and ports(thus neighbors) to generate from this test.

2. Test Controller cleans up compute nodes, and then starts ACA on each compute node.

3. Test Controller creates the containers on compute nodes and assigns IPs/MACs for them.

4. Test Controller sends RESTful requests to Arion Master for the gateway cluster, Arion Wings and also VPCs.

5. Test Controller sends gRPC requests to Arion Master for the neighbors.

6. Test Controller triggers Arion DP Controller’s setup process, which sets up the Arion Wings and other needed metadata.

7. Test Controller starts Arion Agent on each Arion Wing. Arion Agent GETs/WATCHes from the Hazelcast Database, retrieves neighbor information and programs each Arion Wing’s local eBPF maps.

8. Test Controller GETs the gateway information from the Arion DP Controller, and assigns gateways to different subnets accordingly, so that different Arion Wings can handle traffics from different subnets(sharding). Test Controller also assembles the Goalstate with the metadata for the VPCs, Subnets, Ports, Neighbors and Gateways, and sends it to NCM, who will then send the information to each compute nodes. The compute nodes, after receiving the Goalstates, finishes its local setup.

9. Everything is ready, Test Controllers executes pings commands between different ports in the same subnet. The traffic will be directed to the desinated Arion Wing, based on the gateway rules we set up. The ping is supposed to go through.

10. This is the end of the Test Controller.

Example output for Test Controller will be provided below, at the Test Controller section.

For this test, alcor will be run in Docker containers. In order to do so, please follow these steps:

1. Clone the official Alcor repo from https://github.com/futurewei-cloud/alcor.git.

2. Build Alcor, following the instructions here.

3. Get into the scripts folder by cd alcor/scripts.

4. Execute this command to stop and remove existing Alcor Docker conatiners, and build and run them: ./alcor_services.sh -o && ./alcor_services.sh -r && ./alcor_services.sh -b && ./alcor_services.sh -a. Before re-running this end-to-end test, you should restart these services by running this command: ./alcor_services.sh -o && ./alcor_services.sh -r && ./alcor_services.sh -a.

ACA will run on each compute nodes as the host agent. Compiling ACA takes quite some time, if you can make sure all the compute nodes have the same configuration (mainly ubuntu versions), you can compile it on one machine, then copy it to another machine and run it.

1. To compile ACA, please follow this instruction, make sure you run the aca-machine-init.sh as root.

2. To run it, please run this command as root, inside ACA’s directory: nohup ./build/bin/AlcorControlAgent -d -a ${NCM_IP} -p ${NCM_PORT} > /tmp/aca_testing.log 2>&1. Please fill in the correct NCM IP and Port, in order to connect each ACA with NCM. The NCM IP is the host IP of the machine that runs NCM, and the NCM port is 9016, unless you changed it. With the nohup at the beginning, the ACA will run in the background, so that even if your ssh session is disconnected, it should still run. The output of ACA will be directed to /tmp/aca_testing.log, as indicated in the command, feel free to change it if you need to.

3. After ACA starts, run ps aux | grep Alcor to make sure ACA is running, and run ovs-ofctl dump-flows br-tun to see if the default OVS rules exist, below is an example output of OVS rules, when ACA running fine in the background:

Before running the next round of test, make sure you kill the current ACA process, cleanup the environment and restart ACA. Below is a convenient script that can kill the ACA process, remove all docker containers created from the last test, and cleanup the OVS bridges:

NOTE: In the latest test controller workflow, compute node cleanup and ACA starting are included, so users don’t need to do this manually.

Arion Master runs with a Hazelcast Database, and they run as standalone processes in this test. The following are the instructions to download, compile and run them:

Arion Agent is the host Agent of each Arion Wing, like ACA is the host agent of each compute node. You can also consider building it on one of the Arion Wings, and then copy it to each wing, if you are positive that these wing machines share the same configuration. For this test, Arion Agent will be started by the Test Controller, so users don’t run them directly. Please follow this instruction to download and build Arion Agent.

NOTE: In the latest test controller workflow, Arion Agent starting are included, so users don’t need to do this manually.

The Arion DP Controller runs on a Kubernetes cluster, please refer to this page in order to use it.

Notes:

• Before deploying it, please change the hazelcast_ip_port in arion-dp/blob/main/src/mgmt/manager/project/api/settings.py to the actual Hazelcast IP and port. The Hazelcast IP is the host IP of Hazelcast, and the port is 5701.

• After deploying Arion DP Controller, please try to run curl http://${ARION_DP_CONTROLLER_POD_IP}/nodes to check the connectivity. For a freshly deployed Arion DP Controller, you should get the following:

Which is an empty list, as there’s nothing there yet. If you cannot get any valid response, please run command nohup kubectl port-forward svc/arion-manager --address ${DEPLOY_MACHINE_IP} 5000 >/dev/null </dev/null 2>&1 &, to forward the Arion DP Controller’s traffic to ${DEPLOY_MACHINE_IP}:5000. After that, you should be able to connect to the Arion DP Controller:

• To monitor Arion DP Controller’s activity, you can get into its kubernetes pod and monitor its access log and error log. Below are the instructions:

The error.log is very useful when debugging the DP Controller, as it prints logs on almost every step of the setup process.

• Before the next run, please re-deploy the Arion DP Controller.

At last, after all other components are up and running, we configure the Test Controller, and run it to conduct our end-to-end test.

The Test Controller is located at alcor/services/pseudo_controller. To configure, compile and run it, please follow these steps:

1. cd alcor/services/pseudo_controller.

2. Modify the src/main/resources/application.properties all the way until # Test Controller Alcor HTTP APIs Test Params #, those parameters are not used in this test so we don’t need to pay attention to it.

3. Modify the src/main/resources/arion_data.json according to your setup, mainly change the NODE_data section to the configuration of your Arion Wings.

4. Compile the Test Controller in its folder (alcor/services/pseudo_controller), using command mvn clean install.

5. Run the Test Controller using command ` mvn exec:java -D exec.mainClass=com.futurewei.alcor.pseudo_controller.pseudo_controller -e`.

Note: If you need to use the port forwarding for Arion DP Controller, please make sure you set the dp_controller_use_port_forwarding to true.

Attaching an example output below, for users' reference:

Note:

• After running the Test Controller, you can also manually ping between the containers in the same subnet. Simply ssh one of the compute nodes, get into a container, and execute a ping command. Below is an example of it:

• Sometimes you wish to test with a physical network interface with a large traffic, but you only get a fraction of it’s expected performance, you might want to check this page out: https://github.com/futurewei-cloud/alcor-control-agent/wiki/Debugging-TCP-Connection-using-iperf3..