The function of the compute shader was to take the result of the rendering and store it in a compute buffer, so that the RenderTexture could then be re-used. This is roughly what my code looked like:

renderTexture = new RenderTexture(width, height);
ComputeBuffer computeBuffer = new ComputeBuffer(10 * width * height, 4);
copyShader.SetTexture(kernelIdx, "source", renderTexture);
copyShader.SetBuffer(kernelIdx, "result", computeBuffer);
cam.targetTexture = renderTexture;
for (int i=0; i < 10; i++) {
  // Set cam.transform.position/rotation based on i
  camera.Render();
  copyShader.SetInt("startIdx", i * width * height);
  copyShader.Dispatch(kernelIdx, (width  + threadsX - 1) / threadsX,
                                 (height + threadsY - 1) / threadsY, 1);
}
pixels = new uint[10 * width * height];
computeBuffer.GetData(pixels);

The goal is to render 10 images and copy them all into computeBuffer in order. But on some GPUs, the Render() and Dispatch() calls are done out-of-order – sometimes all the Render() calls are done before the Dispatch() calls, resulting in all 10 images in the computeBuffer being identical. Other times, the Dispatch() calls are done just one iteration early or late, shifting the resulting images up or down in the buffer, or resulting in duplication of certain images. I don’t know whether this is a Unity bug or a GPU memory model limitation, but I needed to find a workaround.

I discovered that if I read the entire result buffer after every Dispatch(), this eliminates the issue, but at enormous performance cost:

for (int i=0; i < 10; i++) {
  // Set cam.transform.position/rotation based on i
  camera.Render();
  copyShader.SetInt("startIdx", i * width * height);
  copyShader.Dispatch(kernelIdx, (width  + threadsX - 1) / threadsX,
                                 (height + threadsY - 1) / threadsY, 1);
  computeBuffer.GetData(pixels);
}

Rather than retrieving the entire huge buffer containing all the data, a more efficient method is to introduce a second, very small buffer, and retrieve that instead. Here’s what the modified compute shader looks like:

#pragma kernel Copy

Texture2D source;
RWStructuredBuffer result, forceWaitBuffer;
int width, height, startIdx, forceWaitValue;
SamplerState MyPointRepeatSampler;

[numthreads(32,32,1)]
void Copy (uint3 id : SV_DispatchThreadID) {
    if (id.x >= width || id.y >= height) return;
    if (id.x == width - 1 && id.y == height - 1 && id.z == 0)
        forceWaitBuffer[0] = forceWaitValue;

    float4 color = source.SampleLevel(MyPointRepeatSampler,
                                      float2(((float)id.x + 0.5)/ width,
                                             ((float)id.y + 0.5)/ height), 0);
    color *= 255.0;
    result[startIdx + (id.y * width) + id.x] =
        ((int)color.r << 16) | ((int)color.g << 8) | (int)color.b;
}

The main difference is I’ve added “RWStructuredBuffer forceWaitResultBuffer” and “int forceWaitValue”. Then, I choose just one value of id.x, id.y, and id.z, and for that value I write to the the buffer. Here I use the last value, but this is probably unimportant. On the CPU side I just read this buffer and check the result:

ComputeBuffer forceWaitBuffer = new ComputeBuffer(1, 4);
copyShader.SetBuffer(kernelIdx, "forceWaitBuffer", forceWaitBuffer);
uint[] forceWaitResult = new uint[1];
for (int i=0; i < 10; i++) {
  // Set cam.transform.position/rotation based on i
  camera.Render();
  int forceWaitValue = i;
  copyShader.SetInt("forceWaitValue", forceWaitValue);
  copyShader.SetInt("startIdx", i * width * height);
  copyShader.Dispatch(kernelIdx, (width  + threadsX - 1) / threadsX,
                                 (height + threadsY - 1) / threadsY, 1);
  forceWaitBuffer.GetData(forceWaitResult);
  if (forceWaitResult[0] != forceWaitValue)
    Debug.LogError("Unexpected forceWaitResult value");
}

Checking the result is not strictly necessary but helps avoid situations where the buffer was accidentally not set for whatever reason.

The same technique can be applied to force synchronization of the Camera.Render() call: we don’t want to retrieve all its pixels (which is very slow), but we can quickly retrieve just one of them (in this case the bottom-right pixel), ensuring the rendering has completed:

Texture2D forceWaitTexture = new Texture2D(1, 1);
for (int i=0; i < 10; i++) {
  // Set cam.transform.position/rotation based on i
  camera.Render();
  RenderTexture.active = cubemapRenderTexture;
  forceWaitTexture.ReadPixels(new Rect(width - 1, height - 1, 1, 1), 0, 0);

  copyShader.SetInt("forceWaitValue", i);
  copyShader.SetInt("startIdx", i * width * height);
  copyShader.Dispatch(kernelIdx, (width  + threadsX - 1) / threadsX,
                                 (height + threadsY - 1) / threadsY, 1);
  forceWaitBuffer.GetData(forceWaitResult);
}
RenderTexture.active = null;

In my testing so far this has led to reliable correct performance on any GPU supporting compute shaders, even when taking many images quickly in a row or with very large images; however I have not tested this on all devices and Unity versions (I’m on 5.1.2f1) and there may be limitations that I am unaware of.

The problem

My code was simple enough: render a view to a render texture, then use ReadPixels() to get the result to a Texture2D for further processing:

renderTexture = new RenderTexture(width, height, /*depth*/24);
renderTexture.antiAliasing = 2;
renderTexture.Create();​
// ...
cam.targetTexture = renderTexture;
cam.Render();
RenderTexture.active = renderTexture;
tex.ReadPixels(new Rect(0, 0, width, height), 0, 0);
tex.Apply();​

The output is black. Removing the “antiAliasing” line causes the output to appear correctly in tex, as expected.

When possible: just don’t use ReadPixels

When possible, it’s better to avoid the problem entirely by avoiding ReadPixels(). For example, it’s possible to write a compute shader that directly accesses the anti-aliased RenderTexture on the GPU and manipulates the result in some manner. When possible, this is the most efficient solution, and may be more efficient than what you were already doing.

A complete workaround with a Copy Compute Shader

After stumbling across the finding that compute shaders are able to access the anti-aliased render target, this led to the idea that we can exploit this to get at the data. This will only work on platforms supporting compute shaders; check SystemInfo.supportsComputeShaders and take alternate action such as disabling anti-aliasing if it does not.

First, here is the compute shader, a trivial shader that copies from one render target to another:

#pragma kernel Copy

RWTexture2D dest;
Texture2D source;
int width, height;
SamplerState MyPointRepeatSampler;

[numthreads(32,32,1)] // Must match .cs file
void Copy (uint3 id : SV_DispatchThreadID) {
    dest[id.xy] = source.SampleLevel(MyPointRepeatSampler,
        float2(((float)id.x + 0.5) / width,
               ((float)id.y + 0.5) / height), 0);
}

We use a point sampler and calculate the indices to ensure each pixel is copied exactly.

We create a slot for the compute shader on the script (public ComputeShader copyShader) and assign it to the script by default by clicking on the .cs file in Inspector and dragging the compute shader into the slot.

Next, we will need three things:

1. A render target with antiAliasing > 1, to render our initial scene into with anti-aliasing;
2. A render target with antiAliasing = 1 (of the same size and format), to receive a copy of the scene from the shader;
3. A Texture2D texture to receive the result of ReadPixels().

Finally, here’s what the complete code looks like:

// Initialize textures
renderTexture = new RenderTexture(width, height, /*depth*/24);
renderTexture.antiAliasing = 2;
renderTexture.Create();​
renderTextureCopy = new RenderTexture(width, height, /*depth*/0);
// To allow copy shader to write to it
renderTextureCopy.enableRandomWrite = true;
renderTextureCopy.antiAliasing = 1;
renderTextureCopy.Create();​
tex = new Texture2D(width, height);
copyKernelIdx = copyShader.FindKernel("Copy");

// Render to render texture
cam.targetTexture = renderTexture;
cam.Render();

// Copy from anti-aliased render target to un-antialiased target with compute shader
copyShader.SetTexture(copyKernelIdx, "dest", renderTextureCopy);
copyShader.SetTexture(copyKernelIdx, "source", renderTexture);
copyShader.SetInt("width", width);
copyShader.SetInt("height", height);
int threadsX = 32, threadsY = 32; // Must match shader
copyShader.Dispatch(copyKernelIdx,
                    (width  + threadsX - 1) / threadsX,
                    (height + threadsY - 1) / threadsY, 1);

// Read pixels out of copy
RenderTexture.active = renderTextureCopy;
tex.ReadPixels(new Rect(0, 0, width, height), 0, 0);
tex.Apply();​

Hopefully this’ll be helpful to someone else plagued by this bug. Let me know if you have any questions.

VRCHIVE is a website (currently in alpha) for sharing static 360-degree panoramas and viewing them in VR. I was asked by VRCHIVE to create a Unity script to capture monoscopic 360-degree screenshots in-game (in both traditional and VR games) and upload them to the VRCHIVE website.

In addition to or instead of uploading, it can save the panoramas to disk as image files with equirectangular and/or cubemap projections. It is designed to capture high-resolution panoramas (typically 8192×4096) without causing FPS drops or other issues that are problematic in VR applications, and can be used both by developers and by end-customers who purchase their games.

Install from Unity Asset Store (follow included README to set up)

Samples

• Neos: The Universe
• Space battle game
• Youtopia
• Viking Village Unity Sample (up close with skulls)
• NYC Block #6 (free scene from Asset Store)
• Particle Callbacks Demo – Unity Sample

Other features including stereoscopic capture and GPU-based fast reprojection are planned for the future. Please let me know any questions or suggestions you have, or if you encounter problems!

Download: Mirror 1 • Mirror 2 • Mirror 3

Usage: Create an empty game object, assign the ScreenFader script to it, and adjust the parameters. Leave “Fade in” checked. At runtime, when you toggle the “Fade in” parameter, it will either fade out (when disabling it) or fade in (when enabling it). It can be toggled from scripts, from Playmaker, or via the editor.

It has no resource overhead when the view is faded in, and supports VR and multiple cameras. Note the scene will still be rendered even when faded out (might optimize this later).

The script is partially based on the OVRScreenFade class, but is otherwise released under Unlicense which imposes no requirements whatsoever. Let me know if you encounter any trouble with it or have any questions!

This was a code snippet I did earlier for some work with light field mesh parameterizations. I had a normal vector and wanted to find two orthogonal vectors spanning the plane that the vector is normal to, in order to project another vector into it. This is an underspecified problem, as given one vector, there are many pairs of two vectors that are orthogonal to that vector and each other. This extra degree of freedom can be used to construct a numerically stable procedure that also uses less operations than computing a cross-product.

In the C# code snippet below, n is the input vector and b1 and b2 are the output vectors.

if (Math.Abs(n.x) >= Math.Abs(n.y) &&
    Math.Abs(n.x) >= Math.Abs(n.z)) {
    b1.x = n.y / n.x; b1.y = -1.0; b1.z = 0.0;
    double d = n.x * n.x + n.y * n.y;
    b2.x = n.x * n.z / d; b2.y = n.y * n.z / d; b2.z = -1.0;
}
else if (Math.Abs(n.y) >= Math.Abs(n.x) &&
         Math.Abs(n.y) >= Math.Abs(n.z)) {
    b1.x = -1.0; b1.y = n.x / n.y; b1.z = 0.0;
    double d = n.x * n.x + n.y * n.y;
    b2.x = n.x * n.z / d; b2.y = n.y * n.z / d; b2.z = -1.0;
}
else { // Math.Abs(n.z) >= Math.Abs(n.x) &&
       // Math.Abs(n.z) >= Math.Abs(n.y)
    b1.x = -1.0; b1.y = 0.0; b1.z = n.x / n.z;
    double d = n.x * n.x + n.z * n.z;
    b2.x = n.x * n.y / d; b2.y = -1.0; b2.z = n.y * n.z / d;
}
// Optionally normalize b1 and b2 to unit length here

Derivation

Step 1: Assume for the moment we’re in the first branch where we know n.x is nonzero (we assume at least one component of n is nonzero). For the first vector, b1, we start with this equation (the vectors must have dot product of zero if they are normal):

n.x * b1.x + n.y * b1.y + n.z * b1.z = 0

We substitute values for b1.y and b1.z to simplify the solution for b1.x as much as possible (we set y and z because this results in division by the nonzero n.x):

n.x * b1.x + n.y * (-1) + n.z * (0) = 0
n.x * b1.x – n.y = 0
b1.x = n.y/n.x

Step 2: We find a third vector orthogonal to both vectors by setting up a system of equations based on both dot products being equal to zero. We need three equations, so we also set b2.z to -1, which produces a simple valid solution:

n.x * b2.x + n.y * b2.y + n.z * b2.z = 0, (n.y/n.x) * b2.x + (-1) * b2.y = 0
n.x * b2.x + n.y * b2.y – n.z = 0, (n.y/n.x) * b2.x – b2.y = 0
n.x * b2.x + n.y * b2.y – n.z = 0, n.y * b2.x – n.x * b2.y = 0

I solved this system in Mathematica with this command:

FullSimplify[Solve[{nx*x + ny*y + nz*z == 0, (ny/nx)*x + (-1)*y == 0, z == -1}, {x, y, z}]]

with the solution:

b2.x = (n.x * n.z)/(n.x * n.x + n.y * n.y)
b2.y = (n.y * n.z)/(n.x * n.x + n.y * n.y)
b2.z = -1

Since n.x is nonzero, the common denominator above is also nonzero, and it only needs to be computed once. Note that in Step 1 we did not set both b1.y and b1.z to zero; if we had, here in Step 2 the result would have involved division by n.y, which may be zero.

The same steps apply to find the solution for the cases where n.y and n.z are nonzero, respectively. In each branch only 8 floating-point operations (flops) are required. Even computing a cross-product in 3 dimensions typically requires 9 flops, and this approach finds both orthogonal vectors at once.

Because the formulas involve no subtraction and only divide by the largest component of the vector, the approach is numerically stable for a large range of inputs. However, I haven’t formally analyzed numerical precision, and the addition in the denominator above may result in some loss of precision if the two components have very different sizes.

The procedure is well-suited to CPU-based implementations but may not be the best for a GPU setting such as a shader where branching is expensive. In cases where at least one component is known to be nonzero (and not too small), the corresponding branch can be used exclusively, which eliminates branching.

Let me know if you use the procedure and what you use it for, or if you have any thoughts on ways of improving it!

even if they can figure out a way that wearing them doesn’t make your face melt off and slide down your neck

I… don’t entirely understand this and it sounds kind of terrifying. I’ve worn the Rift DK2 comfortably for over 8 hours while only getting slightly warm, and the Rift CV1 has considerably better ergonomics than DK2, with materials that facilitate free air flow. Comfort won’t be an issue.

You know the 3D TV in your living room, right? Right? Oh wait, you didn’t buy a 3D TV?

VR brings considerably more value to the table than stereoscopic displays – the immersion provided by a wide field of view and good head tracking allows you to achieve the feeling of occupying a space, and to get a sense of scale far surpassing that of stereoscopic TVs. Head tracking and motion controls enable to use your head and hands in a 1-to-1 intuitive direct manner in the space, and 3D audio responding to head tracking allows more precise audio location. Even people with only one functioning eye have reported first-hand VR experiences as being compelling.

people just want to watch TV. They don’t want to sit at the right angle, wear the special pair of glasses, and then sit transfixed at the screen lest their eyes wander to the clock and the effect be lost.

This is another good example of how VR has surpassed 3DTVs. There is no dependence upon angle of viewing, no need to keep your viewpoint fixed, and no crosstalk between channels. Your eyes are able to move freely to any position and each see an independent view.

Then you stop noticing and get on with playing the game, except now to see what’s happening to the left you have to remember to laboriously swivel your entire head, rather than twitch your mouse a bit.

While efficient controls certainly have their place – particularly in frenetic fast-paced games where they let you operate at superhuman speeds – many worthwhile gaming experiences benefit from a more natural interface. Assigning this function to your head is not only more natural and easier to learn for new gamers, but also frees up your controls for other functionality, as in Elite: Dangerous where the VR support enables you to visually track a target and check your control panels while your flight stick operates your ship.

when I’m playing a game on my screen and I want to see behind me, I turn the view around and look behind me, and it kind of feels like I’m inside it.

I’m not going to say monitor-based experiences can’t be immersive or compelling, but the considerable added value of VR for many experiences justifies the minor inconveniences that it brings. VR is not just about being able to look behind you – it’s about immersion, sense of scale, direct user interfaces, eye contact, and many other advantages.

The clamour for 3D has always been generated by tech companies, then carried by an excited audience right up until the first time they try it.

This is true. The clamor for VR, meanwhile, has come from a vocal group of enthusiasts who have not tired of it even after extensive experience with the Rift DK1 and DK2 kits, which were mainly sold to enthusiasts.

BUT HOW AM I SUPPOSED TO KNOW WHO’S IN THE BACKGROUND AND WHO’S IN THE FOREGROUND?! THEY’RE ALL JUST ONE FLAT IMAGE!

It is true that a screen provides many effective cues for depth, such as parallax during camera motion, relative angular size, position relative to the horizon, and so on. People infer a model of a 3D world in looking at their screens. But the added depth cues provided by VR – both stereoscopic and monoscopic, such as parallax during head rotation – provide many additional cues that convey depth much more effectively. Additionally, a screen occupies only a small part of your field of view, particularly vertically due to their aspect ratio, while VR fills a large portion of your vision with the game, allowing you to take it in more quickly.

VR replaces the role of the imagination

VR does not replace imagination – while the world around you feels more immersive and real and more direct interaction with it is available, there are many unresolved limitations that still separate it from a real place, and many in-game references to places that are *not* in your immediate surroundings – after all, people use their imagination when walking around in real life as well.

once your eyes are stinging from sweat, you’re head’s exhausted from carrying the gear, and your dizzy as fuck from constantly spinning about trying to see everything, the idea of a chair and a monitor suddenly becomes very appealing.

The Rift consumer version is both well-ventilated due to its cloth exterior and is also the lightest HMD ever made – although its weight is not yet announced, it’s known to be less than the DK1, which was 380 g. You can barely feel it on your head. And if you’re spinning around wildly in a VR game, you’re doing it wrong – the motion should be subtle and natural, like in real life.

It’s extraordinarily unlikely that Valve or Oculus are going to release their tech at under $200. It would be a huge surprise is Microsoft or Sony did it even on their production scale. A massively expensive peripheral that the vast, vast majority of potential gaming customers won’t ever buy. And without a considerable customer base, any games created needing goggles to be played is never going to see the sales to justify the development.

While the $300-500 range is certainly where the first generation of HMDs is going to be at launch, it’s not remarkably expensive for a new PC peripheral. Sub-$200 HMDs are inevitable within a couple years, between the used units and the cheap knockoffs. While the market may grow slowly, there will still be enough HMD owners that popular VR exclusives with reasonable budgets will be able to reclaim their investment. These titles will sustain the VR industry as it grows.

The issue’s not even that most people won’t want to stumble around in their extremely confined space filled with sharp-edged furniture wearing a cumbersome blindfold, trying to stave off motion sickness long enough to shoot an alien.

Motion sickness is, on the hardware side, pretty much a solved problem. Between effective 6DOF tracking and natural locomotion with standing and room-scale VR, almost no one has reported any motion sickness with the consumer prototypes, including those who had a lot of issues with earlier dev kits. Running into obstacles is also not a serious risk due to software systems like Valve’s Chaperone that let you designate the safe edges of your play space.

publishers will not want to waste hundreds of millions on further VR-only projects, but will support a VR mode for their larger games

Big game projects are driven not only by market concerns but also by creative vision. Every new successful franchise take risks to explore interesting new ideas. By enabling new experiences and modes of interaction, VR offers big opportunities for large publishers who are looking for differentiation. I believe there will be VR-exclusive AAA titles and that they will not only sell enough units to reclaim their investment, but will powerfully drive sales in VR peripherals. In particular, it’s not hard to imagine Valve with their commitment to VR jumping into this space with a VR-exclusive entry in a popular franchise like Portal or Half-Life.

You are correct though that merely tacking on VR support to monitor-based games is not going to fulfill the true potential of the medium.

Eventually developers will stop wasting time putting the VR mode in (much as the brief dalliance with 3D modes for games quickly went away), and games will go back to how they’ve always been, and how everyone has always wanted them to be.

Again, mere stereoscopy doesn’t bring nearly as much to the table as VR, in either immersion or user interaction. As for “how everyone has always wanted them to be”: one look at how gamers respond to popular sci-fi visions of VR gameplay like Ready Player One and Sword Art Online shows that they are excited to play in VR when the tech is ready to deliver a sufficiently compelling experience.

until it can work without a giant headset, making you look like a lost robot tourist, people will always prefer to sit at their screen

The Rift CV1 is anything but “giant” – it can be held easily in one hand, covers only the area around your eyes, and is only 10 cm or so deep. As for looking like a lost robot tourist, when you put your gameplay up on the screen for spectators, people know what you’re looking at and why, and they focus on the game, not fashion.

In short: monitor-based gaming isn’t going away, but VR provides more than enough added value for a variety of compelling experiences to overcome its minor inconveniences. Many of the most important problems to reach a consumer market, like VR sickness and ergonomics, are essentially solved. Its success is not a certainty, but bolstered by a variety of innovative and compelling titles, the dream of VR becoming a new form of media in its own right is no longer unrealistic.

Meanwhile, I got a VR demo going of my 360 panorama capture Unity plugin. You walk around Viking Village in DK2, and at any time can press P to capture a 360 panorama still. It fades to black, captures the screens, fades back in, and then continues processing on CPU for about a minute (converting to equirectangular projection) before saving/uploading. It’s fully asynchronrous so you can play while it’s processing. You can try out the demo here.

I had an issue where as soon as it finished the reprojection processing it would stutter severely for a couple seconds. Eventually I pinned this down to the Texture2D.Apply() call, which I was calling after executing all the SetPixel() calls on my final destination texture which contained the reprojected image. On such a large texture (8192×4096 in this demo) this is a very expensive call, taking multiple frames at the very least. I circumvented the issue by simply writing my result image into a byte array instead of a Texture2D, which is then converted directly into a C# Bitmap via Bitmap.LockBits() (I had to import the System.Drawing.dll assembly, see this post). I can then save out the result to an image file on another thread (EncodeToJPEG() only works on the main thread in Unity, but Bitmap.Save() can run wherever). This has made the experience a lot smoother, with no detectable lag during processing even in the Rift.

I also used my plugin along with some basic replay functionality to produce a short 360 video of some Unity content. I recorded the position of the player in each FixedUpdate() for replay, and used ParticleSystem.Simulate() for making the torches and such go one frame at a time. I used the GPU reprojection which is more resource intensive but about 30 times faster. Even with this it still took about an hour to complete processing for this simply 17 second video – partly because I was capturing everything in 8192×4096 and downscaling to 3840×1920, partly just because there’s a lot more room for optimization. But it’s still super cool to be able to share a complete play environment on YouTube, and to re-play the 360 video in VR in Virtual Desktop. I think this is the beginning of how Let’s Plays of actual full games will be done in VR.

http://alpha.vrchive.com/image/rC

In principle it’s pretty straightforward – I use RenderToCubemap to create a cubemap, then transform it into the equirectangular transform required by VRCHIVE, then upload it using the Chevereto API.

This approach works just fine for small panos, but there are a number of scalability issues as I move to larger panos: the time required for the equirectangular transform grows linearly with the number of pixels, and even an optimized CPU version can take on the order of minutes. I can make it run asynchronously with the game using coroutines, but this slows it down even further.

I created a version of the transform that runs on GPU using compute shaders, which is about 30x faster and can capture a 8192×4096 pano in about 2 seconds on my GTX 980, but these aren’t universally available and I’m encountering some issues where it fails for people on other GPUs (weirdly it works for me both on my fancy GTX 980 desktop PC and my 5-year-old laptop – I may have to try switching to integrated graphics for testing). I believe texture memory consumption may be at fault, as I can reproduce similar failures when Unity’s RAM usage is very high – so I’m investigating various ways to capture the pano using less storage. Unfortunately these too result in processing slowdown.

Transforming the panos to equirectangular on server-side would be more reliable and faster for the player, but also is relatively expensive for a small startup site like VRCHIVE. Another plausible alternative is direct support for cubemap panos in the VRCHIVE viewer.

I haven’t yet addressed the complications raised by e.g. the use of multiple camera rigs with different layer settings, which are prevalent in several VR titles that I know of. I can’t directly call RenderToCubemap on those cameras because they have the wrong settings for it – so I may have to create a separate cubemap camera and clone their layer settings.

Current focus is fixing the failures on other GPUs – luckily my eVRydayVR fan chat has lots of great testers, so I think I can find ways to fix this!

Meanwhile, SVVR is starting tomorrow and I have to prepare for that too. And I just got back this week from the Vive Jam where I got to help with Unity scripting on Electro Smash, a collaboration with several others there (mostly from Virtuix). So many exciting things going on!

• Added a menu to the beginning, four walls around you, where three of them correspond to three sample texts, and the fourth is the splash screen.
• Added animations which smoothly transition from selecting a sample text to reading that sample text. I also show the page dividing into the 3 copies, to help emphasize what is going on (although it may still be not entirely clear).
• Added a pause menu which allows you to view your WPM, go back in the text 1 or 5 lines (as suggested by Metatron testers), or exit back to the menu (which can also be done with back button). Rather than holding the touchpad, you now just tap it to enter the pause menu and tap on the Continue button to continue reading.
• Mitchell from Metatron made me a sweet app icon.
• Renamed the project from VirtuaReader to VR Speed Reader, to better differentiate it from other reading related entries and make its purpose clear.

Main things I learned from this:

• Learned all about how to do simple animations with Mecanim, and sequenced them using state machines. This turned out to be surprisingly complex – I had to set up the animations and animation controllers, set up an idle state and a trigger to move to the animation state, and set up my script to invoke the trigger. In some ways just scripting animations by hand is easier. But it’s a very flexible system (e.g. does blending, manages state machine visually) and worth keeping in mind for the future.
• raycast and colliders are super handy for doing eye gaze selection and menus, but to make it work with Unity world-space UI elements, I need to first add a CollisionBox to them and size it manually;
• I was having trouble with stuttering while loading up the texture resources for the selected sample text. Blair of Technolust fame gave me some helpful advice which works well on Gear VR, which is to fade most of the scene to black except for a small central element, and then load. The time warp artifacts (black at the edges) becomes invisible when this is done.
  To accomplish this, I smoothly translated the selected menu item way back into the distance, to decrease its angular size, while also fading its background color. Then I loaded. Then I restored the default background color.
• I use a trick where I (invisibly) rotate the menu and player together so that the selected item is always in the same location before proceeding with the animation and reading. This simplifies the rest of the task.
  However, I got confused cause Unity told me not to use RotateAround() cause it was deprecated. Turns out RotateAround() was totally the right thing to use – what was actually deprecated was the overload that takes only an axis and a rotation amount. You need to specify a point as well.

This is mostly done and may be my final Jam submission! I haven’t implemented holding back to go to the universal menu but may not bother with this right now. I still need to do a narrated video and a reddit post and dev forum post about it tomorrow. Let me know any thoughts you have!

Video:

Try it yourself! Download apk

Note: this guide is focused on video publishing, rather than streaming, which requires different tools. Since this guide went up the unwarpvr tool got support for Gear VR; see release page.

Summary

• For warped stereoscopic video intended to be viewed in the Rift DK2, directly publish your original 1080p60 recording.
• For unwarped stereoscopic video, which compromises between viewing in the Rift and monitor viewing, record at 1080p60 or higher and then use ffmpeg-unwarpvr to unwarp.
• For unwarped monoscopic video, which provides a view similar to monitor-based titles, play and record the game at 1440p60 (2560×1440 at 60 FPS), then use ffmpeg-unwarpvr to generate the 1080p60 monoscopic view.

Example of unwarped monoscopic video at 1080p60.

Warped Stereoscopic View

In this method, the original gameplay footage is preserved including lens distortion correction and chromatic abberation correction, allowing any viewer with a DK2 or compatible viewer to reproduce your full field of view during play by simply running your video full-screen on their DK2 in extended mode. Note that this may lead to VR sickness for some viewers since they don’t have control over movement. It can also be viewed stereoscopically on a monitor using the parallel or wall-eyed viewing technique, but on a monitor it will appear distorted and with chromatic abberation. This method also has the benefit of distinctively signifying that the content is Oculus Rift content.

For recording settings with the OBS software, see this post. You can, if you wish, add a facecam either in one of the corners, or in the middle bottom without interfering too much with how the image will appear in the Rift. The use of a green screen is recommended to further reduce interference. Facecam in middle-bottom produces the least overlap, but also puts more emphasis on the face.

Sample of chromatic abberation correction from far left at 100%:

Some games (including Titans of Space, Minecrift, Quake II VR, and vorpX) provide an option to disable chromatic abberation correction. This can make the experience slightly better for monitor viewers and slightly worse for Rift viewers. (It is possible to remove chromatic abberation from a video after-the-fact, without affecting distortion, but this is unimplemented right now in unwarpvr.)

Unwarped Stereoscopic View

My method of choice on my own channel, this method provides a compromise between the needs of those viewing in the Rift DK2 and those viewing on their monitors. For those viewing in the Rift DK2, there is more chromatic abberation and more lens distortion than in the warped method, but the content can still be viewed stereoscopically with correct full field-of-view (FOV). For monitor viewers, there is no chromatic abberation, no black space, and no distortion (straight lines remain straight). They can also view the video stereoscopically using the parallel or wall-eyed method.

To use this method, run and record the app normally. For recording settings with the OBS software, see this post. Then download my ffmpeg-unwarpvr tool. Save it to the location where your videos are. Open a command prompt and use “cd” to change directory to the same location. Use this sample command line as a guide, changing “in.mp4” to the name of your video:
ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=1920:1080 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

As shown above, I recommend placing a thin black bar between the two views, which helps visually break them up in very uniform scenes. You can place a facecam in any corner or in top/bottom center – although it will occlude more gameplay content than in the warped method, it will generally not be visible inside the Rift (watch on your Rift to check this). The quality of this method can be slightly improved (especially in the corners) by recording the gameplay at 1440p, as outlined in the unwarped monoscopic method below.

Because the FOV of Rift content is high, you may notice rectilinear perspective distortion (stretching of areas near the corners). The same effect occurs in monitor-based games when raising FOV settings. If you are disturbed by this effect, you can mitigate it by either using the warped stereoscopic method or by using scale_width and scale_height parameters larger than 1, which effectively crop and zoom the video to the central area:
ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=1920:1080:scale_width=1.3:scale_height=1.3 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

However, this will also reduce image quality; to counter this, recording at 1440p is recommended (see below); this was done for the above screenshot. It will also make the content appear incorrectly inside the Rift (the viewer will have correct stereoscopy but will feel as though they are looking through binoculars).

Unwarped Monoscopic View

This method provides an experience similar to watching recorded gameplay of a normal, monitor-based game. It cannot be viewed in the Rift (except using a 2D video player software like Virtual Desktop) and will only be identifiable as Rift content based on subtle cues. The use of a facecam or logo to emphasize that the content is VR is recommended. This approach is the most representative of your real experience in the Rift (where you observe only a single unified view that is focused on the central area), and is the most comfortable to view on a low field-of-view player, like a non-fullscreen video in a window, or on a mobile device.

However, the use of only one eye view massively reduces resolution; as such, we will need to boost the resolution at which we record the gameplay. We walk through the steps of doing this below.

Add 1440p Custom Resolution

To run any game at 1440p, first you need to add a 2560×1440 resolution mode to your Oculus Rift DK2 display. To do this on an NVIDIA card:

1. Make sure the Oculus Runtime has been installed and you have rebooted since installing it. Right-click the Oculus icon in your system tray and choose Rift Display Mode. Choose “Extend Desktop to the HMD”.
2. Switch to Extend Mode by holding the WINDOWS key and pressing P until “Extend” is selected.
3. Open NVIDIA Control Panel.
4. Click Display→Change resolution, then your Oculus Rift DK2 display.
5. Click the “Customize…” button under “2. Choose the resolution”.
6. Click “Create Custom Resolution…”
7. Create a 2560×1440 mode as shown in the screenshot below (if you are flicker-sensitive, you may prefer to use a refresh rate of 75 – I prefer using 60 Hz so that I can reserve more system resources for recording).
   
   Note: Although I use 2560×1440 in this tutorial, any resolution with a 16:9 ratio can be used; the highest I’ve used successfully is 3328×1872. You can also use a lower resolution if you have performance issues at 2560×1440.
8. Click the Test button. Ensure that your DK2 display is displaying correctly and is not black or distorted. Find the confirmation dialog (it may be on your Rift DK2 display) and click “Yes” or press ALT+Y.
9. Right-click your desktop and choose Screen resolution. Click the Oculus Rift DK2 display and then select a resolution of 2560×1440. Then click OK. You must use the Windows settings to do this, not NVIDIA Control Panel.
   

I don’t have an AMD card – if you do and know how to do this on AMD or any other non-NVIDIA card, please tell me how in the comments and I’ll add it here.

Configure Your Game to Run at 1440p

Configure your game to run fullscreen at 2560×1440. The way to do this varies from game to game:

• Most Unity games: Hold ALT while launching the game. The resolution selector dialog will pop up and allow you to choose 2560×1440. In some Unity titles you may have to run them from the command line like this:
  whatever_DirectToRift.exe -screen-width 2560 -screen-height 1440
  
  If the game is running in high persistence mode (bright screen, blur during head movement), exit, run and exit the Tuscany application from Oculus, and then try it again.
• Minecrift: I assume you’ve already set up Minecrift to play at normal 1080p resolution. Next, set your Rift as your main monitor in your Windows resolution settings. Start Minecraft and click “Play game” by looking with one eye in your Rift. Press F11 to go fullscreen. At the main menu click Options→Video Settings→Other and click “Fullscreen Mode” until 2560×1440 is selected.
  
  
  If the game is running in high persistence mode (bright screen, blur during head movement) toggle the Low Persistence setting in VR Settings→Stereo Renderer to fix it.
• Other games: If you’re lucky, some games will simply start up in whatever resolution the display is currently set to. If this doesn’t work, check your in-game menus for any resolution options. Many games unfortunately disable resolution options in VR mode.
  If you can’t figure out how to set up your game for 1440p, the rest of this guide will still work fine, but I highly recommend you use one of the approaches in the “Increasing vertical field-of-view” section below, which will reduce the amount of scaling that occurs during unwarping. If you have a particular game you want me to investigate how to run in 1440p, ask in the comments below!

Set Up Recording for 1440p and Test

My recommended settings for OBS are in this post. To adjust for 1440p, go to Settings→Settings→Video and change the resolution under “Custom”. If your gameplay does not correctly fill the recorded video, you may have to right-click on your capture source under “Sources” and choose Position/Size→Fit to screen.

Test: Make sure you can record a video at 1440p60. Make sure the resolution and frame rate on the video file are correct (right-click→Properties→Details). Take a snapshot of a frame (Video→Take Snapshot in VLC) and zoom in on it in an image editing software. Make sure the quality is reasonable – if it looks blocky or blurry, you may need to look for a setting in your application to increase render target resolution, sometimes referred to as render scale, render texture scale, Oculus quality, or VR quality.

Unwarp to 1080p Monoscopic

Download my ffmpeg-unwarpvr tool. Save it to the location where your videos are. Open a command prompt and use “cd” to change directory to the same location. Use this sample command line as a guide, changing “in.mp4” to the name of your video:
ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=1920:1080:left_eye_only=1:scale_width=1.8:scale_height=1.8 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4
This will generate a view with no distortion or chromatic abberation that completely fills the 1080p frame:

Under close inspection, the image is not perfect – it will have a little bit of “fuzziness”, especially near the edges, and there are subtle “ghosts” from the chromatic abberation:

If your application supports disabling chromatic abberation correction (like e.g. Titans of Space, Minecrift, Quake II VR, vorpX), you can use this option together with the “chroma_in=0” option in ffmpeg-unwarpvr 0.2.2 or later to eliminate ghosting/fringing from chromatic abberation. You will see more chromatic abberation during recording, but otherwise there is no disadvantage. A monoscopic 1080p sample produced using this method:

ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=1920:1080:left_eye_only=1:scale_width=1.8:scale_height=1.8:chroma_in=0 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

Additionally, supersampling can be used to address aliasing artifacts. It works by doubling the output resolution of the unwarpvr filter, then downscaling with a scale filter. Warning: this may greatly slow encoding.  Sample command line (the sws flags improve scaling quality):

ffmpeg-unwarpvr -i in.mp4 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -vf "unwarpvr=3840:2160:left_eye_only=1:scale_width=3.6:scale_height=3.6 , scale=1920:1080" -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

Where the application permits, the chroma_in=0 method can be combined with supersampling for the maximum possible image quality:

ffmpeg-unwarpvr -i in.mp4 -sws_flags lanczos+accurate_rnd+full_chroma_int+full_chroma_inp -vf "unwarpvr=3840:2160:left_eye_only=1:scale_width=3.6:scale_height=3.6:chroma_in=0 , scale=1920:1080" -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

Increasing Vertical Field-of-View

Because the aspect ratio of the view in the Rift is roughly 1:1, cropping it to a 16:9 frame like above results in a dramatic reduction of vertical field-of-view (FOV) that is not representative of your Rift experience.

An approach to mitigating this is to add video content on the left or right sides of the gameplay content, and then reduce the scale_width and scale_height parameters correspondingly. For example, if you add 300 pixels of content on the left-hand side, reducing the resolution to 1620×1080, then the scale factors will be reduced from 1.8 to 1.8×1620/1920 = 1.52. Below is the command line and result (changes in bold):
ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=1620:1080:left_eye_only=1:scale_width=1.52:scale_height=1.52 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

In addition to providing a more representative view, this also decreases the amount of scaling which improves image quality.

Here is a more extreme example with 480 pixels removed from both sides, achieving a ratio of 8:9 which has maximum image quality and vertical FOV and minimal cropping of the recording:
ffmpeg-unwarpvr -i in.mp4 -vf unwarpvr=960:1080:left_eye_only=1:scale_width=0.9:scale_height=0.9 -c:a copy -c:v libx264 -crf 18 -pix_fmt yuv420p out.mp4

When the scale factors get this low, recording at 1440p is no longer essential for achieving good quality; you can record even at 1080p and get decent results.

Finally, if you have two players together in a multiplayer game, you can separately record and transform both their views as in the last example above, and then simply put them side-by-side in your video editing software, as below. The result is similar to the stereoscopic view. Again, for clarity I recommend placing a thin black bar between the two views:

Conclusion

I hope this has been helpful! I’m eager to see the community of VR gameplay content producers grow, and hopefully this will help you all get started and/or improve the image quality of your videos. Let me know in the comments below if you have any questions or suggestions for expanding this article!