Scripts to tar, compress, and upload large datasets to Box. The scripts use GNU Tar's multivolume feature to keep each file's size less than 15 GB and Slurm to parallelize uploading the archives.
- Setup Box for use with curl if you have not done so already
- In Box, create unique password for use with external applications
touch ~/.netrc && chmod 600 ~/.netrc- Edit
~/.netrcsuch that first first line ismachine ftp.box.com, the second line islogin <your_smu_email_address, and the third line ispassword <your_unique_box_password>
- Edit
tar_data.sbatchsuch that<directory_to_tar>is directory to be archived,<archive_prefix>the prefix of the archive files, and<temp_directory>is directory for archive files before upload. - Submit
tar_data.sbatchand wait for archives to be created - In Box, create directory to which archive files will be uploaded
- Edit
upload_data.sbatchsuch that<box_directory>is Box directory for archive files from the previous step,<archive_prefix>the prefix of the archive files, and<temp_directory>is directory for archive files before upload - Submit
upload_data.sbatchand wait for archives to be uploaded