Merge branch 'develop' of github.com:NHSDigital/NRLF into NRL-1876-lloyd-george-perf-test-run

anjalitrace2-nhs · anjalitrace2-nhs · commit b61f0f678d8d · 2026-01-30T14:48:50.000Z
diff --git a/.github/workflows/daily-build.yml b/.github/workflows/daily-build.yml
@@ -67,3 +67,18 @@ jobs:
         with:
           key: ${{ github.run_id }}-nrlf-permissions
           path: dist/nrlf_permissions.zip
+
+  sbom:
+    name: Generate SBOM - ${{ github.ref }}
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Git clone - ${{ github.ref }}
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+
+      - name: Generate SBOM
+        uses: nhs-england-tools/trivy-action/sbom-scan@v1.4.0
+        with:
+          repo-path: "./"
diff --git a/Makefile b/Makefile
@@ -356,7 +356,7 @@ perftest-generate-pointer-table-extract:
 	@echo "Generating pointer table extract with PERFTEST_TABLE_NAME=$(PERFTEST_TABLE_NAME) and ENV=$(ENV) and DIST_PATH=$(DIST_PATH)"
 	rm -rf "${DIST_PATH}/nft"
 	mkdir -p "${DIST_PATH}/nft"
-	PYTHONPATH=. poetry run python tests/performance/perftest_environment.py generate_pointer_table_extract --output_dir="${DIST_PATH}/nft" --extract-size=1500000
+	PYTHONPATH=. poetry run python tests/performance/perftest_environment.py generate_pointer_table_extract --output_dir="${DIST_PATH}/nft" --extract-size=2000000
 	./scripts/get-current-info.sh > "${DIST_PATH}/nft/info.json"
 	zip -r "${DIST_PATH}/pointer_extract-${PERFTEST_TABLE_NAME}.zip" "${DIST_PATH}/nft"
 	aws s3 cp "${DIST_PATH}/pointer_extract-${PERFTEST_TABLE_NAME}.zip" "s3://nhsd-nrlf--${ENV}-metadata/performance/seed-pointers-extract-${PERFTEST_TABLE_NAME}.zip"
diff --git a/poetry.lock b/poetry.lock
diff --git a/scripts/seed_nft_tables.py b/scripts/seed_nft_tables.py
@@ -40,7 +40,7 @@
 
 class TestNhsNumbersIterator:
     def __iter__(self):
-        self.first9 = 920000000
+        self.first9 = 900000000
         return self
 
     def __next__(self):
@@ -111,7 +111,6 @@ def pointer_is_processed(pointer):
             for unprocessed_item in unprocessed_items
             if unprocessed_item["PutRequest"]["Item"].get("id") == pointer_id
         ]
-        # print("unprocessed matches:", matches)
 
         return len(matches) == 0
 
@@ -154,7 +153,7 @@ def _populate_seed_table(
     print(
         f"Will upsert ~{doc_ref_target} test pointers for {patients_with_pointers} patients."
     )
-    doc_ref_counter = 45000000
+    doc_ref_counter = 0
     batch_counter = 0
     unprocessed_count = 0
 
diff --git a/terraform/account-wide-infrastructure/test/dynamodb__pointers-table.tf b/terraform/account-wide-infrastructure/test/dynamodb__pointers-table.tf
@@ -32,6 +32,21 @@ module "ref-pointers-table" {
 }
 
 module "perftest-pointers-table" {
+  source      = "../modules/pointers-table"
+  name_prefix = "nhsd-nrlf--perftest"
+}
+
+module "perftest-pointers-baseline-table" {
   source      = "../modules/pointers-table"
   name_prefix = "nhsd-nrlf--perftest-baseline"
 }
+
+module "perftest-pointers-15m-table" {
+  source      = "../modules/pointers-table"
+  name_prefix = "nhsd-nrlf--perftest-15m"
+}
+
+module "perftest-pointers-55m-table" {
+  source      = "../modules/pointers-table"
+  name_prefix = "nhsd-nrlf--perftest-55m"
+}
diff --git a/terraform/bastion/README.md b/terraform/bastion/README.md
@@ -8,6 +8,7 @@ Before deploying a bastion, you will need:
 
 - An AWS account that has already been bootstrapped, as described in [bootstrap/README.md](../bootstrap/README.md) and has the account-wide infrastructure deployed as described in [account-wide-infrastructure/README.md](../account-wide-infrastructure/README.md). This is a one-time account setup step.
 - Your CLI configured to allow authentication to your AWS account
+- Install the [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
 
 ## Deploying a bastion
 
@@ -17,39 +18,40 @@ To deploy a bastion, you will first need to login to the AWS mgmt account on the
 
 Then, initialise the Terraform workspace with:
 
-```
+```sh
+assume nhsd-nrlf-mgmt
 terraform init
 ```
 
 If you want a read-only bastion (can only READ from the pointers table), plan the deployment like this:
 
-```
+```sh
 make plan-ro
 ```
 
 If you want a read-write bastion (can READ and WRITE from the pointers table), plan the deployment like this:
 
-```
+```sh
 make plan-rw
 ```
 
 Once you're happy with your planned changes, you can apply them with:
 
-```
+```sh
 terraform apply ./bastion.tfplan
 ```
 
 ## Using the bastion
 
 Once the bastion is deployed, you can connect to it via SSH with:
 
-```
-make ssh-connection ENV=perftest
+```sh
+make ssh-connection ENV={env}
 ```
 
 Once connected successfully, you will be at the SSM `$` prompt. To switch to the `nrlf_ops` user, run this command:
 
-```
+```sh
 sudo su - nrlf_ops
 ```
 
@@ -68,7 +70,7 @@ see [user-data.sh](./scripts/user-data.sh) for exactly what's installed on there
 
 If you're running the `make ssh-connection` and are seeing this error:
 
-```
+```sh
 $ make ssh-connection
 ....
 An error occurred (TargetNotConnected) when calling the StartSession operation: i-06ff25164f004bee4 is not connected.
@@ -78,21 +80,23 @@ $
 
 If you've just created a new bastion, it may be that it hasn't started yet. Log in to the AWS console to see the state of the EC2 instance. Press the "Connect" button in the console and choose the SSM tab to see if things are working ok.
 
+If there is a warning in the Session Manager tab "SSM Agent is not online" when you attempt to connect then it's likely the SSM agent has crashed. Reboot the EC2 instance and the SSM agent should start up with previous cli history preserved. To reboot via the CLI, find your EC2 instance > Instance state > Reboot instance. Beware: rebooting the EC2 instance will terminate any ongoing processes.
+
 If the EC2 instance is running and the console looks ok, check you have defined the correct ENV param for the installed bastion.
 
 #### The `nrlf_ops` using is missing
 
 If you're getting this error:
 
-```
+```sh
 $ sudo su - nrlf_ops
 su: user some_other does not exist or the user entry does not contain all the required fields
 $
 ```
 
 If you've just created a new bastion, you may need to wait a little until the cloud-init script has finished. You can check the status of this process with:
 
-```
+```sh
 sudo tail -f /var/log/cloud-init-output.log
 ```
 
diff --git a/tests/performance/README.md b/tests/performance/README.md
@@ -135,7 +135,7 @@ make perftest-generate-pointer-table-extract \
    PERFTEST_TABLE_NAME=nhsd-nrlf--perftest-anjali-test-2-pointers-table ENV=perftest
 ```
 
-This will generate a csv extract of the given pointer table containing a row per pointer. To run the perf tests, you will need an extract larger than the number of test iterations. The default extract size is 1.5 million - this can be changed in the make file command by updating the value of`--extract-size`. Too big and the test runners will take a long time to load the file.
+This will generate a csv extract of the given pointer table containing a row per pointer. To run the perf tests, you will need an extract larger than the number of test iterations. The default extract size is 2 million - this can be changed in the make file command by updating the value of`--extract-size`. Too big and the test runners will take a long time to load the file.
 
 ## Assumptions / Caveats
 
diff --git a/tests/performance/constants.js b/tests/performance/constants.js
@@ -15,9 +15,8 @@ export const POINTERS_TO_DELETE = ALL_POINTER_IDS.slice(0, 3500);
 export const POINTER_IDS = ALL_POINTER_IDS.slice(3500);
 export const NHS_NUMBERS = REFERENCE_DATA["nhs_numbers"];
 
-// filter only 736253001, 736253002, 1363501000000100, 861421000000109, 749001000000101 for now
+// filter only 736253002, 1363501000000100, 861421000000109, 749001000000101 for now
 export const FILTERED_POINTER_TYPES = [
-  // "736253001",
   "736253002",
   "1363501000000100",
   "861421000000109",
diff --git a/tests/performance/perftest_environment.py b/tests/performance/perftest_environment.py
@@ -66,7 +66,7 @@ def __next__(self):
 
 
 def generate_pointer_table_extract(
-    extract_size=1500000,
+    extract_size=2000000,  # must be a multiple of buffer_size (1 million)
     output_dir=".",
 ):
     """
