Skip to content

Lazily download artifacts: The Phantom Menace #27374

Merged
jonahwilliams merged 35 commits intoflutter:masterfrom
jonahwilliams:lazy_download
Feb 8, 2019
Merged

Lazily download artifacts: The Phantom Menace #27374
jonahwilliams merged 35 commits intoflutter:masterfrom
jonahwilliams:lazy_download

Conversation

@jonahwilliams
Copy link
Contributor

@jonahwilliams jonahwilliams commented Jan 31, 2019

Fixes #6491

Only download the minimum*** set of artifacts needed for a particular host, target, and build mode.

*** I still include gradle and fonts in the minimum set.


Adds additional arguments to several Cache methods:

  • buildMode & targetPlatform for filtering by artifacts, can be null. If null, retrieve all artifacts matching one of the other parameters. For example, targetPlatform.ios and null buildMode downloads debug, profile, and release for iOS.
    -skipUnknown, which allows downloading artifacts when targetPlatform and hostPlatform are unknown.
  • clobber, whether to force re-downloading of artifacts

flutter precache will still download the full set of artifacts for a given host platform.
flutter precache -f/--force will re download all artifacts.

Re-organizes engine artifacts into a class from lists of lists of strings.

Flutter commands now check the artifacts for the given or default build mode, and any connected devices. (note: android-arm-profile is downloaded unconditionally because we currently need it for a Flutter doctor check).

This is accomplished via a new method on FlutterCommand called updateCache. This is only called if shouldUpdateCache is true. The default implementation is:

  Future<void> updateCache() async {
    // Only download the minimum set of binaries.
    await cache.updateAll(
      clobber: false,
      skipUnknown: true,
    );
  }

Run, for example, looks at the current set of attached devices and ensures the binaries are downloaded for each kind.

Example doctor command:

jonahwilliams@jonahwilliams0:~/Documents/flutter/examples/hello_world$ rm -rf ../../bin/cache/
jonahwilliams@jonahwilliams0:~/Documents/flutter/examples/hello_world$ flutter doctor
Downloading Dart SDK from Flutter engine 15f2b92cce916982b7dd8ce658bbf2a465c06ba4...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50.4M  100 50.4M    0     0  43.4M      0  0:00:01  0:00:01 --:--:-- 43.4M
Building flutter tool...
Downloading Material fonts...                                       1.7s
Downloading package sky_engine...                                   0.3s
Downloading common tools...                                         2.0s
Downloading linux-x64 tools...                                      1.0s
Downloading android-arm-profile/linux-x64 tools...                  1.7s
Downloading Gradle Wrapper...                                       1.0s
Doctor summary (to see all details, run flutter doctor -v):

BuildMode getBuildMode() {
final List<bool> modeFlags = <bool>[argResults['debug'], argResults['profile'], argResults['release']];
if (modeFlags.where((bool flag) => flag).length > 1)
bool debug;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated getBuildMode to not throw if used in a command that does not add the debug, profile, or release arguments.

if (release) {
return dynamicFlag ? BuildMode.dynamicRelease : BuildMode.release;

if (_defaultBuildMode == BuildMode.debug && dynamicFlag)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were unnecessary since with my change since I moved the check to the top

}

String extraFrontEndOptions =
List<String> extraFrontEndOptions =
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and below was reported to me as a type error.

@jonahwilliams
Copy link
Contributor Author

An additional benefit: we normally spend around 10 minutes downloading artifacts on CI, this PR is spending around 4-5

@jonahwilliams jonahwilliams changed the title Lazily download artifacts [WIP] Lazily download artifacts Feb 1, 2019
@goderbauer goderbauer added the tool Affects the "flutter" command-line tool. See also t: labels. label Feb 5, 2019
@jonahwilliams
Copy link
Contributor Author

Cache issues were due to the engine binary issue

@jonahwilliams jonahwilliams merged commit 98971f3 into flutter:master Feb 8, 2019
@jonahwilliams jonahwilliams deleted the lazy_download branch February 8, 2019 23:24
jonahwilliams pushed a commit that referenced this pull request Feb 8, 2019
jonahwilliams pushed a commit that referenced this pull request Feb 8, 2019
kangwang1988 pushed a commit to XianyuTech/flutter that referenced this pull request Feb 12, 2019
kangwang1988 pushed a commit to XianyuTech/flutter that referenced this pull request Feb 12, 2019
const UpdateResult({this.isUpToDate, this.clobber});

final bool isUpToDate;
final bool clobber;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you document these? It's not clear what clobber represents.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in linked PR

}) {
if (!location.existsSync()) {
return const UpdateResult(isUpToDate: false, clobber: false);
} if (version != cache.getStampFor(name)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting is weird here. Put this on a newline, or put an else in here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in linked PR


/// Hook method for extra checks for being up-to-date.
bool isUpToDateInner() => true;
bool isUpToDateInner({BuildMode buildMode, TargetPlatform targetPlatform, bool skipUnknown}) => true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than having isUpToDateInner(), why not just have subclasses override isUpToDate() and say:

  UpdateResult result = super.isUpToDate(...);
  if (!result.isUpToDate)
    return result;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in linked PR

}

class UpdateResult {
const UpdateResult({this.isUpToDate, this.clobber});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default clobber to false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in linked PR

return false;
return isUpToDateInner();
UpdateResult isUpToDate({
BuildMode buildMode,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildMode and targetPlatform should be properties of the CachedArtifact itself, rather than exist on the update checks, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are passed through so that we can filter down the set of artifacts based on the current known configurations.

@jonahwilliams
Copy link
Contributor Author

@tvolkert I'm addressing your comments in #27735

@jonahwilliams jonahwilliams changed the title Lazily download artifacts Lazily download artifacts: The Phantom Menace Mar 27, 2019
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Would be nice if flutter run only downloaded the necessary artifacts

5 participants