tag:github.com,2008:https://github.com/meroton/bazel/releases 2024-04-16T06:34:01Z tag:github.com,2008:Repository/707213450/7.2.0-pre20240416ros 2024-04-16T07:24:40Z moroten tag:github.com,2008:Repository/707213450/7.0.2-ros 2024-02-01T08:19:55Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/7.0.1-ros 2024-01-22T07:41:47Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/7.0.0-ros2 2023-12-22T16:53:12Z <p>Handle null digest returned by batchStat</p> <p>When RemoteOutputService.BatchStat returns a null digest, the remote <br />output service knows the file exists, but it is incapable of returning <br />the digest. This may be caused by the kernel holding on to dirty pages <br />in its write-back cache.</p> <p>By returning null, ActionMetadataHandler's <br />fileArtifactValueFromArtifact() will compute the file's digest <br />separately.</p> <p>Make sure to run a recent build of bb-clientd to detect these cases.</p> moroten tag:github.com,2008:Repository/707213450/7.0.0-ros1 2023-12-21T07:59:08Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/7.0.0-rc2ros1 2023-10-25T07:31:22Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/7.0.0-pre20231011v2ros1 2023-10-23T14:48:32Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/7.0.0-pre20230926v1ros1 2023-10-20T06:36:58Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> moroten tag:github.com,2008:Repository/707213450/6.4.0-ros1 2023-10-20T06:42:33Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> <p>Change-Id: I320586d7d153627fa2425a54e3cc505befe886c0</p> moroten tag:github.com,2008:Repository/707213450/6.3.2-ros1 2023-10-20T06:40:19Z <p>Remote Output Service: place bazel-out/ on a FUSE file system</p> <p>Builds that yield large output files can generate lots of network <br />bandwidth when remote execution is used. To combat this, we have flags <br />such as --remote_download_minimal to disable downloading of output <br />files. Unfortunately, this makes it hard to perform ad hoc exploration <br />of build outputs.</p> <p>In an attempt to solve this, this change adds an option to Bazel named <br />--remote_output_service. When enabled, Bazel effectively gives up the <br />responsibility of managing a bazel-out/ directory. Instead, it calls <br />into a gRPC service to request a directory and creates a symlink that <br />points to it.</p> <p>Smart implementations of this gRPC service may use things like FUSE to <br />let this replacement bazel-out/ directory be lazy-loading, thereby <br />reducing network bandwidth significantly. In order to create <br />lazy-loading files and directories, Bazel can call into a BatchCreate() <br />RPC that takes a list of Output{File,Directory,Symlink} messages, <br />similar to REv2's ActionResult. This call is also used to create <br />runfiles directories by providing fictive instances of OutputSymlink.</p> <p>To prevent Bazel from reading the contents of files stored in the FUSE <br />file system (which would cause network I/O), the protocol offers a <br />BatchStat() call that can return information such as the REv2 digest. <br />Though this is redundant with --unix_digest_hash_attribute_name, there <br />are a couple of reasons why I added this feature:</p> <p>1. For non-Linux operating systems, it may make more sense to use NFSv4 <br /> instead of FUSE (i.e., running a virtual NFS daemon on localhost). <br /> Even though RFC 8276 adds support for extended attributes to NFSv4, <br /> not all operating systems implement it.</p> <p>2. It addresses the security/hermeticity concerns that people raised <br /> when this feature was added. There is no way to add extended <br /> attributes to files that can't be tampered with (as a non-root user), <br /> while using gRPC solves that problem inherently.</p> <p>3. Callers of Bazel's BatchStat.batchStat() may generate many system <br /> calls successively. This causes a large number of context switches <br /> between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.</p> <p>By requiring that the output path returned by the gRPC service is <br />writable, no-remote actions can still run as before, both with <br />sandboxing enabled and disabled. The only difference is that it will use <br />space on the gRPC service side, as opposed to the user's home directory <br />(though the gRPC service may continue to store data in the user's home <br />directory.</p> <p>I have a server implementation is written in Go on top of Buildbarn's <br />storage and file system layer. My plan is to release the code for this <br />service as soon as I've got a 'thumbs up' on the overall approach.</p> <p>Change-Id: I320586d7d153627fa2425a54e3cc505befe886c0</p> moroten