Skip to content

Releases: spear-sim/spear

v1.0.0

01 Feb 15:53
@mikeroberts3000 mikeroberts3000
v1.0.0
973e3e8

Choose a tag to compare

View all tags
v1.0.0 Latest
Latest

We are happy to celebrate the v1.0.0 release of SPEAR. We are choosing to tag this release as a major version update to indicate our confidence that SPEAR is ready to be deployed in a wide range of projects.

Highlights

control_samples

Figure: The SPEAR Python library can call any C++ function, and can access any C++ variable, on any game entity, and any game subsystem, in any Unreal Engine application, provided the function or variable has been exposed to Unreal's visual scripting system, i.e., Blueprints. We demonstrate this unique capability by controlling 5 embodied agents with distinct action spaces across several Epic Games sample projects: a person and a car from CitySample (top); a flying robot from StackOBot (bottom left); a person with parkour skills from GameAnimationSample (bottom middle); and multiple agents in a resource collecting game called CropoutSample (bottom right). SPEAR can control these agents and perform customizable high-speed rendering without requiring any modifications to the SPEAR library code or the projects themselves, aside from adding a single-line declaration to each project definition file. No further modifications are required to enable comprehensive programmatic control of complex in-the-wild Unreal applications from Python via SPEAR.

hypersim

Figure: SPEAR includes a customizable camera entity that can render a superset of the ground truth modalities available in the Hypersim dataset, and can render 1080p photorealistic beauty images (top row) directly into a user's NumPy array at 55 frames per second. The SPEAR camera can also render fine-grained 24-bit entity IDs that can be used for both material segmentation and object segmentation tasks (middle row), and a non-Lambertian intrinsic image decomposition consisting of diffuse reflectance, diffuse illumination, and a non-diffuse residual term (bottom row).

time_of_day

Figure: By calling existing Unreal Engine C++ functions that are available through SPEAR's general-purpose Python API, it is straightforward to control the lighting in any scene. Here, we programmatically control the lighting to simulate time-of-day changes in the ElectricDreams sample project that is freely available from Epic Games.

pcg

Figure: It is also straightforward to use SPEAR to programmatically interact with Unreal's Procedural Content Generation (PCG) system for dynamically generating scene geometry. Here, we horizontally translate a PCG entity (the big rock structure in the approximate center of each image) across the scene from left to right in the ElectricDreams sample project via a simple Python program. Notice how the main rock structure harmonizes with the rest of the scene in a convincing way (e.g., the water and ground adjusts around the main rock, logs appear and connect with nearby structures), even when it is being driven by simple programmatic control.

metahumans

Figure: SPEAR includes a customizable multi-view camera entity that can render a scene from multiple views at exactly the same time in the Unreal simulation. Here, we render synchronized images from the Metahumans sample project that is freely available from Epic Games.

Feature summary

  • SPEAR can call any C++ function, and can access any C++ variable, on any game entity, and any game subsystem, provided the function or variable has been exposed to Unreal's visual scripting system, referred to in the UE ecosystem as the Blueprint System or simply as Blueprints. There are over 13K functions and over 44K variables that are already exposed to Blueprints (and are therefore accessible in SPEAR) in the UE codebase, and it is trivial to expose new functions and variables by adding a single-line annotation next to a function or variable in a C++ header (see example below).
  • SPEAR provides efficient zero-copy NumPy interoperability, e.g., SPEAR can copy 1080p photorealistic beauty images (see figure above) from the GPU directly into a user's NumPy array at 55 frames per second without requiring any intermediate data copying.
  • SPEAR includes a camera entity that can render a superset of the ground truth modalities available in the Hypersim dataset (see figure above), including fine-grained 24-bit entity IDs that can be used for both material segmentation and object segmentation tasks, and a non-Lambertian intrinsic image decomposition consisting of diffuse reflectance, diffuse illumination, and a non-diffuse residual term.
  • SPEAR can programmatically control standalone shipping games that are already running, live simulations running inside the Unreal Editor, Unreal's path tracer, and the Unreal Editor itself, all through a unified Pythonic interface.
  • SPEAR gives users precise control over how their UE work is executed across UE frames, while also allowing users to execute complex work graphs (i.e., with arbitrary data dependencies among work items) deterministically within a single frame.
  • SPEAR can be used in any Python environment, even on a remote machine, and does not need to be invoked from inside the Unreal Editor.
Assets 2
Loading

v0.5.0

16 Jun 03:41
@mikeroberts3000 mikeroberts3000
v0.5.0
e1165c2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v0.5.0

Precompiled binaries

Changes

This release includes the following new user-facing features and improvements.

  • We have developed an experimental new Python API that can be used to call any public Unreal function, and can be used to get or set any public Unreal property, on any Unreal object, at any time. Our new API exposes more than 10,000 public Unreal functions and more than 50,000 public Unreal properties. As far as we are aware, there is no comparable functionality in any existing video game simulator. We have started using the new API in our mujoco_interop example application (see below), and we expect it to support all of our internal use cases going forward.

  • We have developed an export pipeline to support interoperability with the MuJoCo physics engine. To demonstrate this pipeline, we have developed a new example application, mujoco_interop, shown in the image below. In this example application, an interactive MuJoCo simulation (left) is used to set the poses of objects in the photorealistic SPEAR rendering (right).

image
  • We are releasing 23 new photorealistic Kujiale scenes. The following images of kujiale_0010, kujiale_0020, kujiale_0030 were captured in SPEAR running at over 60 fps.

image

image

image

  • We have made many usability and readability improvements to our code.
Loading

v0.4.0

31 Dec 02:06
@mikeroberts3000 mikeroberts3000
v0.4.0
0bded5e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v0.4.0

Precompiled binaries

Scenes

Windows

macOS

Linux

Changes

This release includes the following new user-facing features and improvements.

  • We are releasing nine new photorealistic Kujiale scenes. The following image of kujiale_0004 was captured in SPEAR running at 4K resolution at 50 fps.

image

  • We are re-releasing our warehouse_0000 scene (pictured below), which is now fully compatible with Unreal Engine 5.

image

  • We have re-enabled our Fetch robot agent, which is now fully compatible with Unreal Engine 5 (pictured above). We have also implemented an improved tutorial, open_loop_control_fetch, demonstrating how to interact with our Fetch agent in a mobile manipulation application.

  • We have made several usability and readability improvements to our code.

Loading

v0.3.0

21 Jul 08:35
@mikeroberts3000 mikeroberts3000
v0.3.0
4e70d6e

Choose a tag to compare

View all tags
v0.3.0

Precompiled binaries

Scenes

kujiale_0000

Changes

This release includes the following new user-facing features and improvements.

  • We are releasing a new photorealistic scene, apartment_0000 under a Creative Commons license. We include all of the Unreal content for this scene directly in our GitHub repository, so users can easily modify and experiment with the content directly in the Unreal Editor. The following image of apartment_0000 was captured in SPEAR running at 4K resolution at 60 fps.

image

  • We have upgraded from Unreal Engine 4 to Unreal Engine 5. This update significantly improves the overall visual quality that users can expect from SPEAR, and makes rendering noticeably more consistent across platforms. The following image shows our kujiale_0000 scene, and demonstrates the visual quality users can expect in Unreal Engine 5. This image was captured in SPEAR running at 4K resolution at 40 fps.

image

  • Under the hood, upgrading to Unreal Engine 5 enables SPEAR to take advantage of Unreal's latest rendering features, including a state-of-the-art fully dynamic global illumination lighting system known as Lumen, and a state-of-the-art virtual geometry streaming system known as Nanite. Lumen enables all lights and objects to be moved fully dynamically with minimal precomputation required, and Nanite enables very high-resolution geometry to be imported and rendered such that the overall rendering cost scales with pixels rather than triangles. We expect these features to be especially useful as we scale the number of scenes that are available in SPEAR in the coming months.

  • We have improved the visual appearance of our OpenBot model.

image

  • We have made several usability and readability improvements to our code, especially to our examples.
Loading

v0.2.0

31 Mar 16:37
@mikeroberts3000 mikeroberts3000
v0.2.0
1d2eaff

Choose a tag to compare

View all tags
v0.2.0

Precompiled binaries

Scene data

kujiale_0000

warehouse_0000

Changes

This release includes the following new user-facing features and improvements.

  • A large new photorealistic scene, warehouse_0000, which can be used to train industrial robots for warehouse applications.

image

  • A new agent, Fetch, that is capable of mobile manipulation tasks.
  • A new example application, open_loop_control_fetch, which demonstrates our Fetch agent performing a simple mobile manipulation task.

image

  • A new scene that is intended for debugging, starter_content_0000, which loads very quickly and is suitable for simple manipulation and navigation tasks.

image

  • Improved behavior when running the SpearSim executable with no additional arguments. The default behavior is now an interactive navigation mode, where a user can navigate around the kujiale_0000 scene.
  • A new command-line tool, run_executable, which launches SpearSim in interactive navigation mode and loads a user-specified scene.
  • An improved communication mechanism between the SpearSim executable and surrounding Python code, which more than doubles the overall frame rate when communicating large image observations.
  • An improved navigation policy that was trained using our imitation_learning_openbot example application to collect data.
  • Our macOS SpearSim executable is now correctly code-signed.
Loading

v0.1.0

10 Dec 00:03
@mikeroberts3000 mikeroberts3000
v0.1.0
14720fc

Choose a tag to compare

View all tags
v0.1.0

Precompiled binaries

  • Windows: SpearSim-v0.1.0-Win64-Shipping.zip
    • If you are running on Windows, you will need to use the executable located at SpearSim-Win64-Shipping/SpearSim/Binaries/Win64/SpearSim-Win64-Shipping-Cmd.exe. This executable ensures that stdout output is routed correctly to the Windows terminal.
  • macOS: SpearSim-v0.1.0-Mac-Shipping.zip
    • Our macOS binary is unsigned, due to a known issue with signing Unreal Engine binaries on macOS. If you are running on macOS, you will need to explicitly grant permission to run this precompiled binary. See this thread for more details.
  • Linux: SpearSim-v0.1.0-Linux-Shipping.zip

Scene data

Loading