FYI @leopardm @pprokic: I created https://github.com/python-gitlab/python-gitlab/issues/3363 that could help here
/cc @nejc
Antoine Auger (29952bb2) at 03 Mar 16:27
feat(gitlab): migrate CI/CD job token allowlists if possible
... and 7 more commits
@pprokic Thanks, let me rebase then
Antoine Auger (d86ce275) at 18 Feb 15:04
fix(gitlab): wait for entities to finish before moving to post-migr...
Hi @pprokic, as always, thanks a lot for the fast review
Also happy to see this fix included in future Congregate 8.x versions, we will be able to get rid of some local patches
Suggestions applied (I tested it again, all good). Back to you
Antoine Auger (3e62f1cd) at 18 Feb 08:08
chore: apply suggestions
Can the status be
created/startedat this point?
No because the post_migration_task is only chained for finished and failed statuses (new logic within this MR):
if entity_status in ["finished", "failed"] and entity_id not in processed_entities:
...
res = chain(
watch_import_entity_status.s(host, token, entity, correlation_id),
post_migration_task.s(host, token, correlation_id=correlation_id, dry_run=dry_run)
).apply_async(queue='celery')Antoine Auger (b78b5317) at 18 Feb 08:01
chore: apply suggestions
Antoine Auger (da9164e1) at 17 Feb 11:06
fix(gitlab): wait for entities to finish before moving to post-migr...
/cc @leopardm
Tested with our usual "E2E tests" (GitLab.com => Self-hosted DEV instance). It works well:
$ cat congregate.log | grep "direct_transfer_migrate"
[17 Feb 2026 10:58:57][INFO]|direct_transfer_migrate.trigger_bulk_import:95| Successfully triggered bulk import request with response: {'id': 128, 'status': 'created', 'source_type': 'gitlab', 'source_url': 'https://gitlab.com', 'created_at': '2026-02-17T11:58:57.108+01:00', 'updated_at': '2026-02-17T11:58:57.108+01:00', 'has_failures': False, 'has_project_entities': False}
[17 Feb 2026 10:58:57][INFO]|direct_transfer_migrate.kick_off_bulk_import:537| Total processed entities: 0, Total discovered entities: 1
[17 Feb 2026 10:58:57][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1209 has status created and is not yet added in the queue
[17 Feb 2026 10:58:57][INFO]|direct_transfer_migrate.kick_off_bulk_import:559| Waiting for all entities to be populated
[17 Feb 2026 10:59:01][INFO]|direct_transfer_migrate.poll_import_status:133| Bulk import 128 still in progress
[17 Feb 2026 10:59:27][INFO]|direct_transfer_migrate.kick_off_bulk_import:537| Total processed entities: 0, Total discovered entities: 2
[17 Feb 2026 10:59:27][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1209 has status started and is not yet added in the queue
[17 Feb 2026 10:59:27][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1210 has status created and is not yet added in the queue
[17 Feb 2026 10:59:27][INFO]|direct_transfer_migrate.kick_off_bulk_import:559| Waiting for all entities to be populated
[17 Feb 2026 10:59:32][INFO]|direct_transfer_migrate.poll_import_status:133| Bulk import 128 still in progress
[17 Feb 2026 10:59:57][INFO]|direct_transfer_migrate.kick_off_bulk_import:537| Total processed entities: 0, Total discovered entities: 4
[17 Feb 2026 10:59:58][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1210 has status started and is not yet added in the queue
[17 Feb 2026 10:59:58][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1211 has status started and is not yet added in the queue
[17 Feb 2026 10:59:58][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1212 has status started and is not yet added in the queue
[17 Feb 2026 10:59:58][INFO]|direct_transfer_migrate.kick_off_bulk_import:559| Waiting for all entities to be populated
[17 Feb 2026 11:00:02][INFO]|direct_transfer_migrate.poll_import_status:133| Bulk import 128 still in progress
[17 Feb 2026 11:00:05][INFO]|direct_transfer_migrate.poll_single_entity_status:327| Entity import for 'siemens-staging' is complete. Moving on to post-migration tasks
[17 Feb 2026 11:00:28][INFO]|direct_transfer_migrate.kick_off_bulk_import:537| Total processed entities: 1, Total discovered entities: 4
[17 Feb 2026 11:00:28][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1209 has status finished and is already in the queue
[17 Feb 2026 11:00:28][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1211 has status started and is not yet added in the queue
[17 Feb 2026 11:00:28][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1212 has status started and is not yet added in the queue
[17 Feb 2026 11:00:28][INFO]|direct_transfer_migrate.kick_off_bulk_import:559| Waiting for all entities to be populated
[17 Feb 2026 11:00:32][INFO]|direct_transfer_migrate.poll_import_status:133| Bulk import 128 still in progress
[17 Feb 2026 11:00:33][INFO]|direct_transfer_migrate.poll_single_entity_status:327| Entity import for 'subgroup' is complete. Moving on to post-migration tasks
[17 Feb 2026 11:00:59][INFO]|direct_transfer_migrate.kick_off_bulk_import:537| Total processed entities: 2, Total discovered entities: 4
[17 Feb 2026 11:00:59][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1209 has status finished and is already in the queue
[17 Feb 2026 11:00:59][INFO]|direct_transfer_migrate.kick_off_bulk_import:555| Entity 1210 has status finished and is already in the queue
[17 Feb 2026 11:00:59][INFO]|direct_transfer_migrate.kick_off_bulk_import:559| Waiting for all entities to be populated
[17 Feb 2026 11:01:02][INFO]|direct_transfer_migrate.poll_import_status:133| Bulk import 128 still in progress
[17 Feb 2026 11:01:05][INFO]|direct_transfer_migrate.poll_single_entity_status:327| Entity import for 'opensource' is complete. Moving on to post-migration tasks
[17 Feb 2026 11:01:05][INFO]|direct_transfer_migrate.poll_single_entity_status:327| Entity import for 'test-project' is complete. Moving on to post-migration tasks
[17 Feb 2026 11:01:32][INFO]|direct_transfer_migrate.poll_import_status:105| Bulk import 128 finished
[17 Feb 2026 11:01:32][INFO]|direct_transfer_migrate.poll_import_status:124| No project entities in this import. Skipping migration results extraction.Just to be safe, this should not happen according to https://docs.gitlab.com/api/bulk_imports/#list-group-or-project-migration-entities
@pprokic This should address your comment, right?
I added a default failed value when the status cannot be retrieved
Antoine Auger (bea82bbe) at 17 Feb 10:47
fix(gitlab): wait for entities to be ready before moving to post-mi...
Closes #1438
Antoine Auger (d05954f1) at 17 Feb 10:25
fix(gitlab): wait for entities to be ready before moving to post-mi...
... and 306 more commits
@pprokic @leopardm I would be happy to get your view on this as it was a bit tricky to implement.
I opted for a "best-effort" approach given that:
Note: this endpoint does not support Fine-grained permissions, I guess this is too recent (GitLab 18.3).
I mentioned it in the customer/gitlab-migration-features-matrix.md