fix: handle deep conflict resolution to preserve non-conflicting files#5873
Conversation
Bug fixes: - filebrowser#5627: Skip inaccessible subfolders in directory listing (was 500) - filebrowser#5294: Cap text editor content at 5MB (prevents memory exhaustion DoS) - filebrowser#5683: Allow copy/move TO root directory (was blocked) - filebrowser#5835: Share creation requires both share AND download permissions - filebrowser#5834: Reject negative/zero TUS Upload-Length, add 10GB sanity cap - filebrowser#5239: Block sharing root directory (too broad, accidental exposure) - filebrowser#5216: Validate JWT signature on token renewal - filebrowser#2078: Support BMP thumbnails, skip GIF resize (preserve animation) - filebrowser#5306: Normalize paths in copy/rename (strip trailing slashes) Ported PRs: - filebrowser#5876: Reject negative TUS upload-length - filebrowser#5875: Check download perm when sharing - filebrowser#5873: Deep conflict resolution — merge dirs, preserve non-conflicting - filebrowser#5832: Optional directory sizes via ?dirSizes=true param - filebrowser#5658: User quota management — configurable quota_resolver callback Config additions: - quota_resolver: callable for per-user disk quotas - max_content_size: cap for text editor loading (default 5MB) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
|
Hi @ArielLeyva, I actually worked on the same issue a few days ago and just submitted a PR for it: #5884. It's unfortunate that I didn't see your work earlier |
|
@ArielLeyva Thanks for your PR. That is a pity that we worked with @geourjoa at exactly the same time! I think that your code has performance issues when you upload a deeply nested structure like: The browser's file input gives you a flat list of files with relative paths. The client iterates over each file and makes an upload request (either POST or TUS) with the full destination path. With skip=true, every single file and folder would hit the server, and the server checks if the destination already exists before writing. This means: If
This is fine for correctness (it works), but it's wasteful when you could short-circuit: if a parent directory doesn't exist, you know nothing inside it conflicts, so you could skip the per-file existence check and just upload everything directly. This a big problem for our use case where hundreds or thousands of nested folders are uploaded. If it is not to much of a burden, could you have a look to @geourjoa draft PR #5884 It implements a tree structure and a recursive function over this tree so it does not check for subfolders if it discovers that a parent does not exist on the server. Also, it adds a "Resume previous transfert" that ignores existing files, except if they are smaller on the server. Indeed, if a big transfert is abruptly interrupter, some files will be partially uploaded and smaller on the server than they should be. I'm not sure but it seems that Anthony's code is more concise (no changes in the backend) and yet more efficient than this proposal. I might be mistaken here and it is not at all to trash talk this contribution that is very welcome :-) |
|
If you are interested in seeing PR #5884 in action, we have a demo here: https://filebrowser-fix.tetras-libre.fr Please comment here and I will send the credentials in DM. |
Hi @daxid, I've been testing @geourjoa 's PR and I think it's excellent. I find it a better implementation than this one in terms of speed, since it's not necessary to upload all the files if you select "Skip All" or the new "Resume previous transfer" option you've implemented. I've been testing it and haven't found any bugs so far. Just a couple of notes. It would be good if, before displaying the conflict resolution modal, instead of making multiple requests for each directory and subdirectory, a single request were made to return the information for all of them (this would make displaying the modal a bit faster). I can work on this while it's still a draft. And the other thing is basically a minor issue: the tooltip that appears in the "Resume previous transfer" option is great, but in the mobile version, its content is difficult to see (I've shown it by long-pressing the button because if I tap it, the "Resume previous transfer" option is selected). Adding the modifier Otherwise, it's great; I think the implementation is solid. |
|
Tank you @ArielLeyva this is great news. Anthony will complete his PR and add your suggestions next week. We will probably have a couple of other PR on the way :-) Go Filebrowser ! |
|
I just pushed new code in my PR :
Have you work on a single fetch ? I'm not so sure if its worth to implement it. The PR will be bigger, more complex to review. We have all tested a lot the current implementation and we are confident on it. And last but not least it, someone must do it. 😆 What's your sentiment on it ? EDIT : Claude has done the fetchAll impl for me, I can start testing ! EDIT 2 : So far no issue, so I make my PR ready to review #5884 |
Excellent, @geourjoa, excellent implementation. I'll take a look and test it out. Regarding the single request improvement, that can be addressed in a future pull request once the current one is approved, keeping the review process simple. |
…lways-reuploaded-even-if-Skip-all-conflicting-files-is-selected
|
I'll wait until you decide whether this PR or #5884 should be merged. |
Description
Previous conflicts were handled in the following way.
For example, considering this file and folder structure:
Upload queue:
lorem.txtipsun/----
dolor.txt----
sit.txtacmet.txtExisting files and folders:
lorem.txtipsun/----
dolor.txtacmet.txtSelecting "Skip all conflicting files" should only upload the
sit.txtfile that doesn't exist in the destination location; however, the backend endpoint doesn't perform a "deep" check and only verifies conflicting files or folders in the root location (it returnslorem.txt,ipsun/, andacmet.txt) as conflicting files, thus emptying the upload queue (since the entireipsun/folder would be deleted).Based on this behavior, there were several problems:
This PR changes the conflict resolution behavior to handle it as follows:
Additional Information
The code for these functions was refactored to make it shorter and more understandable.
Closes #5842
Checklist
Before submitting your PR, please indicate which issues are either fixed or closed by this PR. See GitHub Help: Closing issues using keywords.
masterbranch.