Skip to content

Contaminated S3 remotes #186

@effigies

Description

@effigies

Many templates have multiple S3 remotes, with other prefixes. As a result, exporting to the wrong remote can end up damaging a different template's export tree.

󰅂 git submodule foreach git show git-annex:remote.log | awk '/prefix/{for(i=1;i<=NF; i++){if ($i~/fileprefix/){print $1 " " $i}} }' | sort | uniq -c | sort -k3
      2 505d13e3-0b15-4d78-83aa-9ebd49b3f0b9 fileprefix=tpl-dhcpAsym/
      1 37b2c337-e9c3-43ae-ade5-8b1d68d6b320 fileprefix=tpl-dhcpSym/
      2 c940c9ec-caab-484b-8158-8926b46e0694 fileprefix=tpl-dhcpVol/
      3 61dfa942-0cd2-499c-84fe-44f1cff82fc9 fileprefix=tpl-Fischer344/
      4 1c987f68-a9b8-4beb-9ecf-1d957d1b5cd1 fileprefix=tpl-Fischer344/
      1 544b0f22-cb98-48d5-8f34-5630c0e38c1e fileprefix=tpl-fsaverage/
      2 59e68276-d464-4fa4-b587-215d22964b3b fileprefix=tpl-fsaverage/
      1 171254a4-6c52-4e4b-95fe-ae94848c3734 fileprefix=tpl-fsLR/
      2 200a18a6-53f7-412a-ab3e-8cbda4ea0743 fileprefix=tpl-fsLR/
      2 b6f9c9fd-394c-488a-939b-ce99cbfc9f61 fileprefix=tpl-fsLR/
      1 60b4188c-8fa0-48e0-a168-ff736658bdf4 fileprefix=tpl-MNI152Lin
      5 60b4188c-8fa0-48e0-a168-ff736658bdf4 fileprefix=tpl-MNI152Lin/
      5 c921da3e-a12a-46c1-9d43-05c6591826a5 fileprefix=tpl-MNI152Lin/
      6 9f919012-4b36-4137-a788-7b002a4288f7 fileprefix=tpl-MNI152NLin2009aSym/
      3 05dfa999-38c6-4a64-97d4-4f9a49a9561d fileprefix=tpl-MNI152NLin2009bAsym/
      5 3d82b038-eeb8-4a84-ac4b-3adf37a51b2b fileprefix=tpl-MNI152NLin2009bSym/
      1 d7f1d2b9-afae-47ab-8d51-19fee65c97c3 fileprefix=tpl-MNI152NLin2009cAsym/
      2 5b090303-8bb9-4689-b438-e4e3c2b9191a fileprefix=tpl-MNI152NLin2009cAsym/
      1 603dfb2c-0a0b-42e2-803e-04078a9738fc fileprefix=tpl-MNI152NLin2009cSym/
      1 bb95249c-b6cb-4ff9-a801-93041a403abe fileprefix=tpl-MNI152NLin2009cSym/
      2 31a8f640-48a8-4d47-bbbb-28eee51eda70 fileprefix=tpl-MNI152NLin6Asym/
      2 be06b1c2-7d35-4bcb-8c30-88e1a1b687fd fileprefix=tpl-MNI152NLin6Asym/
      1 6c24b4dc-99e8-404d-a0c3-9491860d8226 fileprefix=tpl-MNI152NLin6Sym/
      1 80461790-3292-49f8-9e35-0fa8727563cd fileprefix=tpl-MNI152NLin6Sym/
      1 9ac08cff-dd81-42e3-adee-c086dcc4d4a8 fileprefix=tpl-MNI305/
      1 ab3a7553-a511-474b-97ea-acf67df49e70 fileprefix=tpl-MNI305/
      1 e973dee1-c68b-4ae3-9c59-8d6ba85c0279 fileprefix=tpl-MNIColin27/
      3 5a3ce762-e5e9-472d-863f-b66f56da25be fileprefix=tpl-MNIInfant/
      3 f409c791-ce48-45bc-bd63-d657a6263b82 fileprefix=tpl-MNIInfant/
      1 0398ebe5-42e5-4c65-b5ba-8071f2abffde fileprefix=tpl-MNIPediatricAsym/
      1 96a37f83-6703-4158-9e08-ffec44f0b497 fileprefix=tpl-MNIPediatricAsym/
      1 ad339b27-4b22-477c-905f-7b894f89a5cd fileprefix=tpl-MNIPediatricAsym/
      1 c3500d35-6177-4d09-834f-c83b33c8490f fileprefix=tpl-MNIPediatricAsym/
      1 1036f8e3-8e69-4f03-995b-dec9fd71e9f3 fileprefix=tpl-MouseIn/
      2 c8600450-3417-4f4c-91aa-9fc25ae13806 fileprefix=tpl-MouseIn/
      2 9e1f586a-b1cf-4bd4-841a-a5e64016244a fileprefix=tpl-NKI/
      2 c771fd8b-e9f0-4d2a-9bee-2c24c2c7b456 fileprefix=tpl-NKI/
      1 591e58e8-8a23-483c-b456-c542d5789da9 fileprefix=tpl-NMT31Sym/
      2 6a0d5bd5-ba76-4795-a8eb-ec9a83f8b6b3 fileprefix=tpl-NMT31Sym/
      1 d0df914e-736a-483e-8132-b22948593264 fileprefix=tpl-OASIS30ANTs
      1 61dfa942-0cd2-499c-84fe-44f1cff82fc9 fileprefix=tpl-OASIS30ANTs/
      1 ccb18c6b-306a-4181-96d8-bad0ce3e0e94 fileprefix=tpl-OASIS30ANTs/
      5 d0df914e-736a-483e-8132-b22948593264 fileprefix=tpl-OASIS30ANTs/
      6 8b220e24-2a59-456e-8e02-8e83b675525e fileprefix=tpl-OASIS30ANTs/
      1 05dfa999-38c6-4a64-97d4-4f9a49a9561d fileprefix=tpl-onavg/
      1 bef4aa66-01b5-4d13-9ad9-2dae8e407e6c fileprefix=tpl-PNC
      1 a21746d9-e9e6-4d66-820d-9eec630a529d fileprefix=tpl-PNC/
      1 bef4aa66-01b5-4d13-9ad9-2dae8e407e6c fileprefix=tpl-PNC/
      1 034e4660-e89e-4207-8711-9dd47493dfcb fileprefix=tpl-RESILIENT/
      1 d5673afc-2bb3-4c0d-85b1-e6f8a5c21367 fileprefix=tpl-RESILIENT/
      1 a06d0e39-40e2-414a-9ae8-963e35fa3e4f fileprefix=tpl-SUIT/
      1 63f78e8e-55eb-4998-9d1d-c7e75a886aca fileprefix=tpl-UNCInfant/
      1 8b3da2dd-6992-4d0c-94fd-74a3979a8438 fileprefix=tpl-UNCInfant/
      1 5dce00a4-b3b8-4da7-9956-92477ab84cfe fileprefix=tpl-VALiDATe29/
      1 ed33c0cc-1469-4446-8d24-dbb2c69c0a6b fileprefix=tpl-VALiDATe29/
      1 8a2dc774-8f9f-46d7-99db-6fc8e467c379 fileprefix=tpl-WHS/
      1 ee91e717-70b7-476c-8d42-376ed09f17ad fileprefix=tpl-WHS/

In some cases, the same remote UUID shows up in multiple repositories (e.g., fileprefix=tpl-dhcpAsym/), in other cases, there are multiple remotes with the same prefix (fileprefix=tpl-WHS/).

We need a canonical remote for each template, ensure that remote is only activated in the correct template, and disable it in all other templates. It would also be good to figure out how this pollution is happening.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions