Turn back on macOS shard Cirrus caching#50496
Conversation
jmagman
added
c: contributor-productivity
jmagman
force-pushed
the
cache-cirrus
branch
from
dc32884 to
f600493
Compare
|
No more dart SDK, pub, or artifacts download during setup. Also reduces setup time from 3 minutes to 1:30 minutes. |
.cirrus.yml
Outdated
There was a problem hiding this comment.
Is there some way we can prevent this from breaking? Maybe a test that checks that this is the only file changed?
There was a problem hiding this comment.
Why would this file invalidate the cirrus cache? Aren't we fingerprinting bin/internal/*.stamp?
There was a problem hiding this comment.
Ahhh.. do we use this in the tool to say we don't trust the cache?
There was a problem hiding this comment.
I know the shell wrappers don't check it. It looks to me like this is just used for how often we show the user about a newer upstream version.
There was a problem hiding this comment.
Other than making it readonly, I can't think how. bin/cache is gitignored so there's no way to tell what's changed in the working directory. Creating a sub-repo in that directory in setup to then reset before upload would take minutes to init and commit because of the large binaries.
#44526 would make this better.
There was a problem hiding this comment.
Why would this file invalidate the cirrus cache? Aren't we fingerprinting
bin/internal/*.stamp?
The cache invalidation doesn't come from the fingerprint, it comes from Cirrus's hashing of the cache. From https://cirrus-ci.org/guide/writing-tasks/#cache-instruction
After the last script instruction for the task succeeds, Cirrus CI will calculate checksum of the cached folder (note that it's unrelated to fingerprint_script instruction) and re-upload the cache if it finds any changes. To avoid a time-costly re-upload, remove volatile files from the cache (for example, in the last script instruction of a task).
There was a problem hiding this comment.
ahhh, i see. thus any changes of the cache during execution will trigger a re-upload.
.cirrus.yml
Outdated
There was a problem hiding this comment.
@Hixie @dnfield Do either of you know why the other shards have separate Cirrus caches for pkg and artifacts in the bin/cache but not that entire directory? I don't think the Linux or Windows caches are getting used since all the stamps are missing and it just re-downloads everything again.
Example:
https://cirrus-ci.com/task/6607166220009472?command=setup#L420
There was a problem hiding this comment.
I think that's just how they were always set up - it does look like we're getting cache hits (https://cirrus-ci.com/task/6607166220009472?command=flutter_pkg#L16) but maybe failing to use them in setup...
It seems likely to me that this worked at some point and we fixed something in the tool that broke it.
There was a problem hiding this comment.
Yeah, even worse than ignoring the cache, we're downloading it, then the tool blows it away because the stamp files aren't there and re-downloads them. Should this PR include linux + windows?
There was a problem hiding this comment.
I was going to do Windows and Linux in a separate PR. I want to get the mac changes out the door ASAP with Cirrus's network flakiness.
|
Talked offline - I think the only way to address my concern that makes sense is to split readable and writable cache as Jenn is suggesting. LGTM |
|
Holding this until I get clarity from @fkorotkov about what seems to be cache corruption at https://cirrus-ci.com/task/6396945455382528?command=all_flutter#L17 that's causing cache misses. |
jmagman
force-pushed
the
cache-cirrus
branch
from
74ec9b9 to
25879c4
Compare
jmagman
force-pushed
the
cache-cirrus
branch
from
925e2dd to
36ad793
Compare
jmagman
force-pushed
the
cache-cirrus
branch
from
36ad793 to
352b09d
Compare
|
This reverts commit e002698.
4 participants
Description
Revisiting #49585 in light of revent Cirrus network unstability on mac VMs.
Related Issues
Reverts part of #37880.
Checklist
///).flutter analyze --flutter-repo) does not report any problems on my PR.Breaking Change