docker-icloudpd/change.log at master · Whob1/docker-icloudpd

728 lines (436 loc) · 41.3 KB
 - Change: Synology systems create multiple directories which contain extended attributes, such as thumbnails. These will be seen as "eaDir*" directories. New variable added to skip changing permissions on these directories. PR submitted by tsanie
 - Change: Rewrite of remot sync function. Should be a little bit more reliable, as the previous one would get differeing results back from the Telegram API. Now it goes by update ID and stores it's progress to a file, so that it doesn't fail to retrive a correct value at boot.
 - Change: Added function to remove deleted files from Nextcloud
 - Change: Re-enabled **delete_after_download* functionality as recent fix has been merged upstream
 - Change: Altered the clean up of the *notification_title* so that it does not apply to the commonly used Chinese notification methods
 - Change: New function to Nextcloud server. This allows uploading directly to a Nextcloud server, rather than just dropping a file for a different container to monitor and then start a manual sync
 - Change: Rebuilt with iCloud Photos Downloader 1.14.2
 - Change: Rebuilt with Alpine Linux 3.18.2 and iCloud Photos Downloader 1.13.4
 - Change: Just chown the tmp and configu folders as some people seem to be having issues with keyring file which is solved by moving the file from the true location to the /config dir location. Possibly a permissions issue. Not sure.
 - Change: The Telegram polling is still acting weird. Sometimes, the latest message reported by the telegram API will actually be an older message, then a second call to Telegram's API will return a higher number. Also, after a while, all messages will just disappear, meaning null values are passed back. Hopfully these should be fixed now.
 - Change: New variable added **albums_with_dates** which should allow albums to be downloaded to folders with the album name, and then split by date.
 - Change: Send notification when remote sync command received
 - Change: Out of range issue polling for Telegram wake command
 - Change: Added **skip_album** variable so that certain albums (such as the Apple default ones) can be excluded from the download.
 - Change: Unexpected behaviour when latest message is an update about group membership. This change will set ${latest_message} to ${current_message} if ${latest_message} is empty
 - Change: Now supports downloading from multiple folder albums, and saving them into folders named accordingly.
 - Change: Bug fix for folder structure
 - Change: Possible solution for Telegram polling issue
 - Change: Set the function to configure file ownership to use user and group ids (instead of names) as this was failing to set the correct permisisons prior to the accounts being created (brand new containers).
 - Change: So, I messed up and accidentally pushed a version I was working on, to Dockerhub and to Github, which contained my Telegram API token. I had to delete the latest tag from Dockerhub, reset the master branch on GitHub and rotate my Telegram token. What a pain in the arse. That means this latest version is a little rushed, so don't be surprised if it's broken. It does bring within it a cracking new feature though... I've now coded it so it polls the Telegram chat (the one it normally sends notifications to), and if the latest message is the same as the **user** variable, it will force an immediate sync. That way, if you've taken a load of photos on your phone, you can tell your NAS at home to sync the queue once you're done.
 - Change: This is just a change to the documentation. There has been a variable for Telegram **telegram_silent_file_notifications** which sends the telegram notifications for download/delete previews silenetly, I just noticed it's not been documented so I thought I'd change that.
 - Change: I've re-enabled the **delete_after_download** option as it's been implemented correctly upstream and newly released.
 - Change: I've also removed the i386 build platform from my images. They're failing to compile as they can't build a wheel for cryptography. I'm hoping nobody complains, but it they do I'll look at trying to re-add it in.
 - Change: So many changes, I've forgotten most of them... Um... now I use Python virtual environments to install mutliple versions of icloud-photos-downloader... It incorporates v1.12.0, which is used for downloading photos globally. It also uses a modified version of v1.7.3 which allows authentication against the China servers. Also rebased the image to use Alpine 3.17.3. 
 - Change: Changed any "True/False" variable to "true/false" and also find/replace title cased variables in icloudpd.conf to lower case
 - Change: Some stuff around how build tags are handled in docker buildx. Mainly so that I have versioning in Dockerhub, instead of only a single "latest" container
 - Change: Delete after download option removed due to bug where it deletes files from iCloud that have not been downloaded yet
 - Change: New build refers to --china-mainland rather than --domain cn
 - Change: FFS, I typo'd it as --china_mainland
 - Change: Rebuilt using icloudpd and integrating PR 608 https://github.com/icloud-photos-downloader/icloud_photos_downloader/pull/608 in an attempt to fix the on going China authentication issue
 - Change: Got rid of auth_china variable, as it doesn't work.
 - Change: Altered the way the log file creation works so that it can add missing/future variables and sort it into alphabetical order. 
 - Change: Urgh. It's filth, but it might just work... Docker container downloads a forked copy of icloud_photos_downloader which has been customised to work with China's authentication servers. The cookie generation function will now swap out/in these versions of the files if the auth_china variable is set to True, or just set really, I'm not fussy on this one
 - Change: Added a --enabledebugging and a --disabledebugging command line option which will edit the config file to enable/disable debugging for people who may find editing a file a little trickier on some systems... I'm thinking NAS devices may not make this so easy
 - Change: Made it so that the script reads the value of the debug_logging variable prior to each sync. That way debug logging can be switched on using the command line and it will apply on the next run
 - Change: Complete overhaul of logging system. A DEBUG log now introduced which removes a lot of the standard logging and puts it behind this option. When it's enabled, it will hide some of the sensitive data such as e-mail address, Apple ID and tokens
 - Change: Complete overhaul of variables. Docker variables are now pushed into the icloudpd.conf log file stored in the config directory. This allows users to toggle options by just changing the log file and restarting the container. This will allow people to switch on/off deug logging, or change the notification method without having to remove and recreate the container
 - Change: The --ConvertAllHEICs function was missing a command to create the destination folders. This broke the cuntion is being used in conjunction with the jpeg_path variable
 - Change: Rebuild to Apline Linux 3.17.2 and icloudpd 1.12.0
 - Change: Added delete_after_download variable. This moves all downloaded files to 'Recently Deleted'. Cannot be used with auto_delete variable
 - Change: Added support for Telgram proxy server using the variable telegram_server
 - Change: Added function to delete all JPGs that have an accompanying HEIC. Use --RemoveAllJPGs to remove that data. Be warned, JPGs that have matching names with a HEIC file in the same folder will be removed. They may not necessarily be the same photo. Use with car and only if you know what you are doing
 - Change: Added details in readme warning users about enabling Apple's new Advanced Data Protection feature in iOS 16.2
 - Change: Changed the way permissions and ownership is applied so that it it only applies permissions to the download directory after a sync, rather than at multiple different times.
 - Change: Slight change needed to user creation due to rebase to Alpine 3.17
 - Change: Added --Init so I'm less likely to typo it
 - Change: When downloading from libraries it now lists available libraries to the log file
 - Change: The iOS16 shared libraries patches have been moved to a new container tagged as "testing", something isn't quite right as many people are reporting errors.
 - Change: iOS 16 shared libaries can now be downloaded with the --library option. PR courtesy of Patrice Neff  https://github.com/pneff
 - Change: Added support for WeCom proxy via PR submitted by Alano-i and Sowevo
 - Change: Incorrectly named variable. changed bark_key to bark_device_key
 - Change: Amended the log entries so that the warning about setting icloud_china variable only appears if it's not already set. Also changed it to prompt the user to log into iCloud.com and make sure there are no pop-up notifications.
 - Change: Bug in --RemoveKeyring preventing it from launching
 - Change: Documentation updated, changed --DeleteKeyring to --RemoveKeyring
 - Change: Added feature to place converted JPEGs into different folder from the download location
 - Change: Keyring file wasn't actually being removed during container initialisation... Bizarre. Always thought it was. Maybe I removed it while ago, but forgot. Added a --DeleteKeying command line switch to remove the keyring file, if it exists.
 - Change: Bark modifications
 - Change: Amended error relating to unindexed an library. Error says to wait minutes, it should say hours.
 - Change: Added Bark notifications (untested)
 - Change: Two fixes required for Python 3.10+ support
 - Pull request: Merged PR from rmlanghopto to add Gotify notifications
 - Change: Set the function, which corrects the owner and permissions, to run after a successful download. Now permissions are corrected immediately after files are downloaded, rather than at the start of the next scheduled synchronisation.
 - Change: Moved the Nextcloud trigger to the donwloaded/deleted files notification sections (as it's essentially notifying Nextcloud that it needs to sync), but also because where it was currently executing, meant that it would only trigger a sync on successful downloads, not if files had simply been removed. Now it will trigger a Nextcloud oneither of those two events.
 - Change: Added trigger_nextlcoudcli_synchronisation variable.
 - Change: Changed the notification events so that they can have a larger range of icons and different ones too.
 - Change: Gone live with some changes to WeCom notifications.
 - Change: Amended the find command to ignore symbolic links when changing the group on downloaded files and their folders. This is due to the Synology Photos app creating symlinks in the download folders.
 - Change: Partially added some Chinese translations, still a few questions about some other changest that have been requested, so holding off on all of them.
 - Change: Cookie expiration is not working for expired cookies, only the notifications about expiry. This change should change the icon in the notification to the alert symbol when cookie has actually expired.
 - Change: Reverted the changes that were put in place to support root user. Running as root user is really bad, and I probably should have just sacked it off when it was requested, so I'm putting it back to how it was because some people using unRAID devices have seen a load of issues, because they're actually running the thing as root... Why even is that?
 - Change: Evaluate the home directory location of the user... If user is root, home is located at /root. On CoreElec/LibreElec/OpenElec, then the user's home directory is in /storage. This change should work for either of those scenarios.
 - Change: Allow badnames when creating user id.
 - Change: UNTESTED BETA: Added WeCom, the official method of WeChat notifications.
 - Change: Add option to download from single album.
 - Change: Bug in logic about 2FA renewal reminders means they aren't being sent out daily. I've changed logging to help identify where the problem lies.
 - Change: Added single_pass variable which runs the sync process once before exiting out.
 - Change: Merged PR from  eliempje (https://github.com/boredazfcuk/docker-icloudpd/pull/155) which adds Openhab functionality.
 - Change: Apparently Synology devices have a Photos app which doesn't update when a download takes place. The newly created files need to be touched for it to trigger a re-index. I've added a new variable which will touch a file to trigger the update, then set it back to what it was to maintain integrity.
 - Change: Added a check for the IP address lookup performed during initialisation. Previously this could fail if the DNS server was not available (due to container starting on same system as a DNS container). It will now retry the lookup for 2mins before failing.
 - Change: Changed cookie generation name from Generate2FACookie to GenerateCookie as it generates both types of cookie and is run no matter what cookie type you use. Also, changed the log output so it doesn't say "Generating 2FA Cookie" when generating a Web cookie.
 - Change: Urgh. I dunno about you, but sometimes I delete a bunch of stupid memes off my phone and it ends up leaving my folder_structure with empty directories. I've now added a variable delete_empty_directories which will tidy those up.
 - Change: The timezone variable is not being set globally if it's not specified, so the underlying Python script generates a warning. The TZ variable is now defaulted to UTC in the Dockerfile which should take care of this.
 - Change: Added an extra synchronisation interval of 21600, which synchronises every 6 hours. This generates a warning message detailing the potential issue and fix, and includes a 2 minute wait on container start time to make the warning prominent.
 - Change: Pushover notifications not showing downloaded/deleted file previews.
 - Change: Added function to allow deletion of HEIC accompanying files. These would be the converted JPG files (if using the HEIC to JPG conversion feature) and the_HEVC.MOV files which come with live photos.
 - Change: Bug found with sending Telegram notifications if e-mail address has an _ character in it. The _ character wasn't escaped correctly when passing the text to the Telegram API.
 - Change: The Notify function is always called, even if notifications aren't configured. A log entry was still being processed and the logic for determining a success/failure of the sent message was being performed too, resulting in an 'out of range' error, as an empty variable was being processsed.
 - Change: Cookie expiry notification period was being defaulted in the Initialise function and also for every notification type configured in the ConfigureNotifications function. Removed these as they're just duplicates. Also corrected a bug which would override user configured period, configuring it to 7.
 - Change: Logic for skipping the file check was checking if value it true, then following two if statements had to check if it was set to false. Don't really like the OR tests that needed to be done so adjusted the logic two set the check_exit code and check_files_count to values that will always perform a sync, as they should never change due to CheckFiles function never being run.
 - Change: Script doesn't seem to reliably send the "once per day" notifications during the nofitication expiry period. So have added a log entry which details the next notification time
 - Change: Huge overhaul of notification system. Probably broken everything lol. I've tested Telegra, Prowl, Discord and Pushover. I don't have any of the other methods to test though, so can't guarantee they work, or are formatted correctly. I've standardised the way notifications are sent to the Notify function. This way, only one notification command is needed to send notifications for all types and the formatting of the messages is now handled by the Notify function.
 - Change: Introduced a bug with my error handling from the last update. Now I create the error and exit_code log files in the Initialisation function of the script so that the correct permisions are applied. Problem was that the files were being created as root, and then the python script didn't have permission to them.
 - Change: Status code checking for the curl command when sending notifications would only accept 200 as a success. Discord reports 204. I've amended the checking code to just accept 2xx.
 - Change: Handle errors differently. Added suggestions for authentication failure issues.
 - Change: Eliminated full stops from log entries. Some lines had them, others didn't, so standardised on not having them. We don't need them. They take up precious bytes. They giving me flashbacks to COBOL.
 - Change: Broke out logging into three separate functions. Just because. I dunno. Having tonnes of $(date '+%Y-%m-%d %H:%M:%S') commands in the script just looked a bit weird. Plus if I even need to change it, it's three lines instead of a find/replace of a bazillion.
 - Change: Set ConfigureNotifications function to check for presence of Dingtalk token, instead of ID, to decide whether to fail or not. Same as the other methods (except webhook, which doesn't use tokens).
 - Change: Altered priority for Pushover warning/error notifications. Previously they were set to -2, which gives no alert, only changes the icon badge number. Now it sends a standard priority alert for all notification types, except the one that alerts you that your cookie will run out that day, which is a "quiet hours breaking" notification (same for Prowl).
 - Change: Amended the code which sends to Pushover notifications. Originally this was two separate commands, depending on whether the pushover_sound variable was sent. This is not needed, as Pushover will use the default sound is a blank value is sent for the "sound" argument
 - Change: Changed notification failure log messages to be more accurate and display http status code.
 - Change: Prowl notifications logging not quite right.
 - Change: Standard notification log message.
 - Change: Moved some stuff to prevent duplication of code.
 - Change: Added variable to set custom sounds for Pushover notification type. This can be used to silence notifications.
 - Change: Fix bug introduced by moving to useradd/groupadd.
 - Change: Amended domain and route checks to be compatible with icloud_china variable. No point checking the route to icloud.com is OK when the container downloads from icloud.com.cn.
 - Change: Change to IYUU notifications, which should now get these working, and integrated changes to live.
 - Change: Users and Groups now created with useradd/groupadd instead of adduser/addgroup to allow the creation of users/group with an id greater than 256,000.
 - Change: Final text alert for cookie expiration not received. Made a couple of changes. Will check again in 3 months time.
 - Change: Added the IYUU notification type. This is a 3rd party service which supports forwarding messages to WeChat.
 - Change: Added variable icloud_china so that photos are downloaded from icloud.com.cn instead of icloud.com 
 - Change: Removed readme information about obsolete variable interavtive_only. The readme is nearing it's maximum size on Dockerhub, so this helps keep tthe number of characters down.
 - Change: Amended healthcheck so that it does not set the status to unhealthy if notification days is less than 7. This was causing my auto-heal container to automatically restart it every 5mins. This behaviour now only occurs when the cookie has actually expired.
 - Change: Made script launch parameters case insensitive and also allowed for non-English spelling of Initialise.
 - Change: Set icloud.com DNS query type to 'a' records only so that it returns the IPv4 IP address and not IPv6. Also, removed the local loopback addresses, just to remove more unwanted data
 - Change: Discord notifications added by @ibtvt
 - Change: Added detection for when a file has been downloaded, but also exists in 'Recently Deleted', so gets removed post download. In these cases, it should get rid of the missing file errors.
 - Change: Added DNS lookup and traceroute to check network connectivity to icloud.com
 - Change: Removed the code which removes hyphen characters from the Telegram chat ID. The hyphen character is used to denote the bot has been added to a group to send messages, rather than a standard user chat.
 - Change: Replicated recent whitespace changes to additional functions for completeness
 - Change: Force convert HEICs functions now remove pre-existing JPG files as I don't think the convertion tool is overwriting pre-existing files
 - Change: Bug introduced by recent HEIC to JPEG change. Fixed.
 - Change: Added set owner and permissions function to run after HEIC to JPEG conversion functions
 - Change: Added --ForceConvertAllmntHEICs
 - Change: Fixed bug in above new command line option as it wasn't handling spaces safely
 - Change: HEIC to JPEG coversion utility has a bug and rotates JPEG files when it shouldn't. Replaced with ImageMagick which doesn't have this problem.
 - Change: Added --ForceConvertAllHEICs command line option to overwrite JPEG files. This is so over-rotated files can be replaced with crorrectly oriented versions.
 - Change: Added a log entry to show the UID of the account that is running the script. It seems some users are having permissions issues and this will help with debugging
 - Change: Also changed the symbolic link command to force linking of python_keyring directory if something that isn't a link exists in its place.
 - Change: Added logging for --ConvertAllHEICs function
 - Change: Amended way script handles expired cookies
 - Change: Amended script to wait for the cookie file to be created, then continuing once it's appeared. Container will auto restart after 30mins if file does not appear, just in case something has gone wrong.
 - Change: Better way of calculating elapsed time.
 - Change: Amended the logging so it lists the expiry time of expired 2FA cookies for clarity.
 - Change: Amended the way the script waits for the keyring and failsafe files to be created. Instead of checking every 5 minutes before proceeding, script now checks every 5 seconds. This way, script is a bit more responsive when being initialised. If the script doesn't detect the file it's checking for after 30 minutes, it will exit the container, just in case there is a detection problem.
 - Change: Added logging info explaining that the download check takes a long time on large collections and that nothing is being downloaded during that phase.
 - PR Merge: Changed how the webhook notification is handled for compatibility across more notification platforms.
 - Change: Script now records time taken per synchronisation and subtracts this from the synchronisation_interval. This means a sync with a 12 hour interval will start 12 hours later and not skew by the amount of time it takes the sync to complete.
 - Change: Locked version of tzlocal to 2.1 as lateat version introduces dependency which is not available in Python 3.8
 - Change: Added variable manipulation on the Telegram chat ID to remove minus sign if it's included in the variable. It needs to be omitted when posting the data to the API.
 - Change: Added some extra logging info
 - Change: Added function to sanitise command line parameters passed to script.
 - Change: Download and Delete notification webhook payload JSON fix
 - Change: Remove ambiguity in DownloadedFilesNotification
 - Change: Sanitise notification_title variable
 - Change: Rebase to Alpine Linux 3.14.0
 - Change: Modify 20210709 JSON escaping and fix minor change.log text
 - Change: Removed apple_password variable. Password must now be added to the system keyring.
 - Change: Default mode now to download, interactive_only variable removed. This process was far more complex than it needed to be for Synology/QNAP devices, so a small change to the default operation makes things easier for everyone.
 - Change: --Generate2FACookie command line option changed to --Initialise as it is also used for saving the password to the system keyring. Not just cookie generation
 - Change: Changed JSON escaping to prevent further issues.
 - Change: Download check and actual download functions both exit with same error message. Changed for clarity about which section is failing.
 - Change: JPEG quality mismatch in documentation and script. Default JPEG quality now referred to correctly in README.MD as 90.
 - Change: Added script hash to make sure that the sync-icloud.sh file is correct file as date stamp seems to be affected by timezone.
 - Change: Oddities occuring when creating password/cookie on devices which always run interactively
 - Change: Accurate health check output when approaching cookie expiration date.
 - Change: Cookie expiration date checking improved.
 - Change: Wasn't setting notification period for some of the notification options.
 - Change: Notifications weren't firing in the notification period. Should now be fixed.
 - Change: Code to remove empty keyring file requires logic to detect f file actually exists, otherwise brand new containers enter a file deletion loop. Logic amended.
 - Change: Rebase to Alpine Linux 3.13.5
 - Change: Now logs full version ID rather than just major/minor version.
 - Change: Reddit user Vinnipinni made a comment that the documentation was great... so I've revisited it again to make sure it's up to date, corrected some spellings and grammar, and made a few improvements for clarity.
 - Change: --size parameter was incorrectly set as --photo-size.
 - Change: Added code to remove keyring file if it doesn't contain credentials
 - Change: New download link for Boost source as location changed
 - Change: Amended log output for when attempting to add a user to a pre-existing system group.
 - Change: Amended log text after cookie generation. Script no longer claims 2FA cookie generated for all cases, but states 2FA/Web based on which was actually generated.
 - Chnage: Add Litecoin address. BTC network fees are expensive AF!
 - Change: Added notification_title variable.
 - Change: Readme changes to add full stops on lines where they were missing.
 - Change: Added logging of $TERM variable.
 - Change: Modified when the script check for the the --Generate2FACookie so that it can be invoked from NAS devices such a Synology and QNAP.
 - Change: Added the --Generate2FACookie command line option.
 - Change: Rebuilt on Alpine 3.13
 - Change: Convert HEIC to JPEG functions now allow specifying the quality with and integer value from 0 to 100.
 - Change: Sanitise apple_id variable so all letters are lower case. This prevents a case mismatch between the cookie file name that is saved, and the name that the icloudpd Python script creates the cookie file with, due to it respecting case.
 - Change: Removed Pushbullet notifications. The Pushbullet service has introduced version 2 of its API so notifications no longer work. Pushbullet have also removed their app from the iOS App Store, so if it's no longer supported on iOS, there's no way to realistically support it here.
 - Change: Upgraded pip
 - Change: Pushover notification bugfix. PR courtesy of @larstomas
 - Change: Webhook notification notification standardisation. PR courtesy of @larstomas again.
 - Change: Altered failure piority for pushover notifications from -2 to 2 so they become emergency level notifications rather than silent. Change recommended by @larstomas
 - Change: Added variable to allow skipping of the new file check. On large photo collections, this can significantly increase the time taken to download.
 - Change: Added log line to show the exact command line used to invole iCloudPD
 - Change: Due to changes in iOS14 the 2FA synchronisation interval is now limited to a maximum of once every 12 hours. It seems that 2FA authentication is now restricted to a maximum of 26 logins before re-authentication is required. Synchronisation is now restricted to specific time periods with the shortest being 12 hours.
 - Change: New variable synchronisation_delay added. This is so that the first synchronisation can be dalayed for a number of minutes. This is so that multiple containers can stagger their downloads
 - Change: Bugfix. Convert HEIC to JPEG function would only run if download notifications were enabled.
 - Change: Additional changes required for Pushover notifications.
 - Change: Added Pushover notifications
 - Change: Failsafe file mount checking changed slightly regarding logging
 - Change: Added variable for the --until-found option as must have missed this on the 20201024 update.
 - Removed: Checking downloaded files list against files stored on the server is no longer required. Issue has been fixed upstream.
 - Change: Bugfix on healthcheck not detecting non-zero exit code.
 - Removed: Multi-thread capability has been removed from upstream so removed capability to specify number of threads
 - Removed: Speed test capability. Was put in to test speed differences between mutli-thread and single-thread download options. No longer possible due to previous change.
 - Change: Bugfix. Convert HEIC to JPEG function - missed a bit.
 - Change: Bugfix. Convert HEIC to JPEG function failing to set correct timestamp.
 - Change: Remove pip cache folder /root/.cache during container build.
 - Removed: Recently files only doesn't work as expected. Removed.
 - Change: Added --CorrectJPEGTimestamps command line option to correct timestamps of JPEG files
 - Change: Convert HEIC to JPEG functions don't preserve timestamps. Filestamp of JPEG now taken from HEIC file.
 - Change: Default to downloading new files only by checking what files exist vs what needs downloading. This prevents it attempting to download all files upon every synchronisation.
 - Change: Typo for Synchronisation Interval variable. This change will break users configurations that set the old variable name, so default to 24hr sync period for these users.
 - Change: Silence warning about depreceated command_line_options varaible if it's actually blank to begin with.
 - Change: Remove Python package Future. This is for Python 2 to Python 3 compatibility, however, upstream application now fully Python 3 compatible.
 - Change: Python keyring and keyrings.alt no longer version restricted. Upstream application is now compatible with newer Python versions than before so these older packages no longer need to be installed.
 - Change: Bugfix relating to Prowl & Pushbullet notifications.
 - Change: Added new variables: set_exif_datetime, auto_delete, photo_size, skip_live_photos, live_photo_size & skip_videos. Eventually command_line_options variable will be removed.
 - Change: Added a new "CommandLineBuilder" function to create the command line options list to be passed to the download command.
 - Change: Added a speed test mode which ignores the requirement for a mounted filesystem. It limits the download to last 500 files.
 - Change: Multi-threaded mode can now be enabled by setting the multi_thread variable. This will set the thread count to the number of available processors, multipled by five. If the variable is not configured, it will default to single threaded node. Please note: Multithreaded mode has known issues withe files that have duplicate file names. Use with caution.
 - Change: Lowered 60 second delay for warning about password not being stored in keyring to 15 seconds as 60's a bit long.
 - Change: Removed the check for files that have been downloaded already, as new icloudpd version uses photo fingerprinting to allow downloading of different pictures with identical names.
 - Change: Added two variables, download_notifications and delete_notifications. If they are not set, they will default to True. To disable, set to False.
 - Change: Dockerfile now installs pip dependencies using requirements.txt instead of having them specified in the dockerfile.
 - Change: sync-icloud.sh script now shows the version of Python, icloudpd and pyicloud-ipd that are being used.
 - Change: Noted that folder structure can now be set to 'none' to download to flat file structure.
 - Remove: diff file that applies the patch to skip incorrectly named files. Pull request has now been merged.
 - Change: Undersocres now being used in cookie file name. Amended to reflect change.
 - Change: Made the webhook path configurable.
 - Change: Removed the Powershell script to build separate images for ARM architectures as I'm now producing a single multi-arch image without using Windows.
 - Change: Version is now date and time of last modification of the sync-icloud.sh script.
 - Change: Added force_gid variable so that user can be added to system groups to maintain consistency with host OS.
 - Change: Rewrote the functions to create the user/group as it would fails if a system reseved group id is chosen.
 - Change: Amended the interactive terminal check to see if /dev/stdin is a pipe. Odd issue seen on some systems.
 - Change: Updated READ.ME.
 - Change: Updated READ.ME.
 - Change: Apply diff patch to fix album naming issue raised here: https://github.com/ndbroadbent/icloud_photos_downloader/issues/150.
 - Change: Removed manual versioning in Dockerfile in favour of MD5 script hash. Means I don't need to track version numbers, or remember to increment them, but latest version can be confirmed quite easily from the log.
 - Change: Added owner change to /config dir at start of ConfigurePassword function so password is successfully stored first time around.
 - Change: Converted docker-compose.yml file to be 3.8 compatible. Removed networking configuration.
 - Change: When using keyring it was claiming plaintext password was in use and vice versa.
 - Change: Added a version number in Dockerfile to display when script first starts up.
 - Change: Found that the --directory option is required. I wasn't including it when doing the 2FA authentication cookie generation which lead to an error.
 - Change: Rebased to Alpine 3.12.
 - Change: Display IP Address in log.
 - Alpha change: Changed Webhook notification messages.
 - Alpha change: Forgot to add webhook_id to the check which validates that the API keys have been set.
 - Alpha change: Webhook host_ip_address variable changed to webhook_server. Added webhook_port so a non-default port can be added.
 - Alpha: Webhook notifications implemented for Home Assistant. Based on Telegram notifications, but implemented without testing. Using them will probably break things as I'm unsure as to the format of the webhook payload.
 - Change: Prefixed some of the files stored in /tmp/icloudpd with icloudpd_, just for consistency. Plus it's easier to remove them all with `rm icloudpd*` instead of multiple arguments to rm.
 - New: Keyring password saving. Previously, the Apple ID user's password could be seen in the Docker logs, the variables (docker inspect) and disturbingly, if any user did `ps -ef | grep password", it would be seen in the command line of any running connection to iCloud. Script now heavily pushes users towards saving credentials into the secure encrypted keyring.
 - Removal: Saving password to text file has been removed (never made it to a release, but is doumented in the change log).
 - Change: su doesn't pass variables to the account it uses. This meant that some variables that I thought were optional (user, config_dir, folder_structure) were actually mandatory. Command now built with $0 $1 $2 etc and variables passed via command line ( -- $bunch $of $options).
 - Removal: Debug logging variable removed. Always logging errors to screen.
 - Change: Telegram notification success/failure messages now confirm message type (startup, download, delete, failure).
 - Change: Cookie checking function would hang system without error if 2FA cookie does not exist.
 - Change: Container now saves the Apple ID password to a file located in the /config volume. Once this has been done, the apple_password variable not longer needs to be passed to the container and should be removed.
 - Change: Moved some of the logging commands so they triggered in more appropriate places.
 - Change: File download check error notification incorrectly labelled as a startup notification.
 - Change: Script wasn't properly handling passwords with special characters. Should be OK now.
 - Change: Amended the failsafe error message so that the name and path of the file it is looking for is visible. Makes it easier for people to spot the capitalisation in iCloud.
 - Change: File check exit code grabbing logic was flawed. Was actually grabbing the exit code of su, which was always 0.
 - Change: Previous change didn't work. This actually fixes the issue that the list of files is no longer growing. However, it's only highlighted a different issue. A number of files are exhibiting weird behaviour. 5 files are being listed as downloaded, but only 3 filenames are unique. These files are then also being listed as having been deleted from the system. They are all stil in my photostream on iCloud.com and not in recently deleted. Coincidentally, they all start with camphoto_ but that may not be indicitive of anything.
 - Change: My list of downloaded/deleted files seemed to be getting longer without me actively using my phone's camera. Guessing some variable isn't being cleared, so have localised a fair few variables to the functions they run in.
 - Change: Added $debug_logging variable, which outputs errors to the screen instead of hiding them.
 - Remove: Removed PR 132. PR 132 was based off PR 110, which was raised in 2018. Neither had been merged. Obviously, after I spent an hour or so chainging my Dockerfile to incorporate the patch, it was merged upstream!
 - New: Added PR 132: "Multi-threaded downloads and caching" from https://github.com/ndbroadbent/icloud_photos_downloader/pull/132.
 - New: Added quotes to variables to prevent globbing in download check and photo download functions.
 - New: Added interactive_only variable. Some hosts only run containers interactively (looking at you Synology) and this means the script gets stuck attempting to create a 2FA cookie every time. Setting interactive_only will force the script to bypass the cookie generation function and sync files instead.
 - New: Log now displays OS version.
 - Removal: Removed feature to delete JPG file if the HEIC has been removed. Floored logic in that all files exist in a single location, which doesn't match their original location. Also, JPG files received by WhatsApp would be removed as would JPG files saved from Safari. Can't work in practice.
 - New: Script now has a command line parameter that can be run from an interactive session. --ConvertAllHEICs will scan all HEIC files and convert them to JPG, provided the JPG doesn't exist already.
 - Change: Convert HEIC to JPEG not working correctly.
 - Change: Running the sync client with "--only-print-filenames" actually creates folders too. File check now run as user account instead of root to avoid ownership issues.
 - Change: Fancy Telegram notifications caused the underscore characters in the file previews to be displayed in italics. Should be correctly escaped now.
 - Change: Odd issue where owner on config directoy changed to root. Added chown at start of sync to fix this, and also to Owner/Permissions setting function.
 - New: Fancy pants Telegram notifications with emojis.
 - Removal: Telegram URL Encoding function. Now sending markdown text and UTF-8 for special characters.
 - Change: Renamed $syncronisation_interval variable to $synchronisation_interval.
 - Update: Now based on Alpine Linux 3.11.
 - New: Apple ID displayed on push notifications.
 - Update: ARM images rebuilt to latest version.
20200122 - Changes:
 - New: HEIC to JPEG conversion.
 - New: JPEG removal if original HEIC  removed (experimental, backup your files!).
 - Change: Code changes for readability and simplicity.
 - New: Added example docker-compose.yaml.
 - Change: Variable name changes for readability and naming accuracy.
 - New: Pre-download check for new files so a download run will only occur if new files exist.
 - New: Telegram notifications.
 - New: Synchronisation summary. Number of new files downloaded. Number of deleted files (if --auto-delete enabled).
 - New: Telegram notifications only: List of downloaded filenames (10 max). List of deleted files (10 max).
 - New: Startup notification.
 - New: Configurable permissions on the download destination directory.
 - Change: Logic re-writes for simplicity and optimisation.
 - Change: Additional logging.
 - Change: Code clean-ups.
 - Change: Healthcheck update.
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

change.log

Latest commit

History

change.log

File metadata and controls
