Verify and redownload corrupted .nupkg files#552
Merged
Martin-Molinero merged 3 commits intoQuantConnect:masterfrom Mar 3, 2025
Merged
Conversation
893e5fa to
89c67d7
Compare
89c67d7 to
e75427e
Compare
Collaborator
Author
|
@jhonabreul Let me know if you have any other comments and the next ticket to be resolved. By the way, Slack is having issues, I can't send you a DM |
Collaborator
|
----------------------
Next one doesn't have an issue created, but this is what we need:
You can download data using the cli with the `lean data download` command.
See for instance this dataset
<https://www.quantconnect.com/datasets/algoseek-us-equities> to download
equity data (click on CLI for examples), you can download trade daily data
for AAPL and MSFT equities with this command:
lean data download `
--dataset "US Equities" `
--data-type "trade" `
--ticker "AAPL, MSFT" `
--resolution "daily"
Some datasets are on a "pending" state (see the QCDataset class) which
means we haven't made it public yet. For instance the "US Future Universe
<https://www.quantconnect.com/datasets/quantconnect-us-future-universe>"
dataset is pending. If you try to download data from that dataset, the cli
will tell you it can't find it.
For example, run lean data download then select "QuantConnect" and then
"Financial Market Data" and make sure "US Future Universe" is not listed
(because it's pending).
What we want is to have an env variable and a constant to enable listing
pending datasets (see for instance the `API_BASE_URL` constant in
`constants.py`). It should default to false. If true, you should see the
"US Future Universe" dataset (and eny other pending datasets) listed.
Let me know if you understand or ask me anything
…On Wed, Feb 26, 2025 at 3:42 PM JosueNina ***@***.***> wrote:
@jhonabreul <https://github.com/jhonabreul> Let me know if you have any
other comments and the next ticket to be resolved.
By the way, Slack is having issues, I can't send you a DM
—
Reply to this email directly, view it on GitHub
<#552 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGAMAWBYZXFVDAEIJS4P5R32RYKLRAVCNFSM6AAAAABX5NVSOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBWGAZDKOBYGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
[image: JosueNina]*JosueNina* left a comment (QuantConnect/lean-cli#552)
<#552 (comment)>
@jhonabreul <https://github.com/jhonabreul> Let me know if you have any
other comments and the next ticket to be resolved.
By the way, Slack is having issues, I can't send you a DM
—
Reply to this email directly, view it on GitHub
<#552 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGAMAWBYZXFVDAEIJS4P5R32RYKLRAVCNFSM6AAAAABX5NVSOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBWGAZDKOBYGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
dee5be3 to
dd96984
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This update ensures that downloaded .nupkg files are not corrupted before use. If a file is found to be corrupted, it is automatically deleted and re-downloaded to maintain integrity. The verification is performed using zipfile.testzip(), which checks for issues in the ZIP structure of the NuGet package
Closes #505