EnviroLight Meets HoloLens 2

It’s hard to believe that it’s been 4 years since our first HoloLens hacks. Some of those early events are still the most memorable to me. It was deeply meaningful seeing peoples eyes light up as we learned together how to bring code and art into the real world.

It was at one of these early events that I first met Angel Muniz. Angel had just finished her Bachelor of Fine Arts degree where she specialized in Environment and Hard Surface modeling. Angels skills proved invaluable at helping her team reach high frame rates, but we couldn’t help noticing how much more beautiful her models appeared even on Sketchfab than they did on the device. Angel and I wanted to understand why this was happening, but more importantly we wanted to learn how to make holograms appear as lifelike as they could be. Thus, EnviroLight was born.

The Challenge

We knew from the beginning that our issues had mainly to do with light (hence the name). Angel was using Physically-Based Rendering techniques, but we identified 4 main issues when deploying to HoloLens:

1. Limited Shader Support – The HoloToolkit shader in 2016 didn’t support PBR slots like Roughness or Ambient Occlusion. Though the Unity Standard shader did support them, it wasn’t optimized for HoloLens.
   
2. Light Sources – Light sources have a tremendous impact on everything we see. Light sources control obvious things like shadows and colors, but they also control less obvious things like our perception of bumpiness and even our perception of temperature. On HoloLens, many apps ship with a single white directional light. While this gets the job done, it almost never represents the real world space where the application is running.
   
3. Reflections – Without reflections it’s almost impossible to see how shiny or rough an object is. But what can we reflect on a HoloLens? If we reflect anything other than the actual environment it could break the illusion or possibly even become a distraction.
   
4. Holograms are Light – And since they’re made of light they don’t react to light like other objects do. They don’t blend into their surroundings because they’re not affected by shadows. In fact they tend to stand out for being too saturated or too bright.

EnviroLight – Our First Attempt

In 2016 we really only had the tools to solve issues 1 & 2. Since our models were highly optimized and we were maintaining good frame rates, we decided to use Unity Standard shader. This did have a significant impact on our performance, but since our goal was realism we accepted the unfortunate penalty.

For light sources … well, we cheated. We left it up to the user, requiring them to point at the surface mesh and say “Add Light” for each bright source in the room. This command created lights wherever the user was pointing, and though we didn’t capture color or brightness the effect was still miles ahead of anything we’d seen at the time.

That version of the app still looks great even today, and it remained on GitHub in it’s original form until June 2020.

EnviroLight Redux

Though I wanted to refresh EnviroLight for some time, a lot had changed in 4 years! HoloToolkit became MRTK, and that update introduced significant changes to the input system. HoloLens 2 also brought new methods for interacting with hand tracking and direct manipulation.

I needed a catalyst to get me motivated on the update, and that motivation came in the form of MR Lighting Tools.

The moment I saw MRLT I just knew we needed to adopt it. Together with PBR support in the new MRTK Standard Shader, we now had a solution for every single one of our issues!

• Reflections – MRLT dynamically generates a reflection map from your own environment using the camera on the device. This means that holograms can actually reflect the room that you’re in!
  
• Light Sources – The reflection map that MRLT generates is a HDR skybox. This means that not only does it generate reflections, it also generates dynamic lighting for each of the light sources in the room! This map represents the color and brightness for each light, and MRLT even configures a Unity directional light for accurate bump maps and shadows.
  
• Holograms are Light – While this problem can never fully be resolved, because MRLT generates reflections and lighting that are accurate to the space, holograms appear much more blended with their surroundings. If the environment gets darker, the holograms get darker too. This is because their virtual surfaces are “reflecting” light from the same light sources in the physical room.

Try EnviroLight Today!

For the rest of this article I’ll cover how I updated EnviroLight to HoloLens 2. If you’re not interested the details or simply want to try it out right away, here you go!

Download App Package (for HoloLens 2)
or
Get the Source

No HL2? No problem. You can watch the video or check out the animated clips below.

In this first clip, notice the ability to pick objects up and touch UI elements on the screen.

In this night clip, notice how light from the TV reflects off the desk and note that light from the bottle illuminates the real world too!

EnviroLight sure has come a long way! 😊

Migrating from HL1 to HL2

Many developers upgraded from HoloToolkit to MRTK a while ago, but if you still need help with that please check out Upgrading from HTK. In this article, I’ll focus mainly on the new input mechanics for HoloLens 2.

Holding Objects

HoloLens 1 supports hand gestures, but interactions are done at a distance. You might look at an object from a few feet away, then air tap to interact with it. HoloLens 2 still supports this model, but since HL2 supports fully articulated hand tracking there’s an exciting new focus on direct manipulation. The classic HL1 model is now referred to as “Far Interactions” and the new HL2 model is referred to as “Near Interactions”. Adding near interactions was a fair chunk of the update, but thankfully MRTK made this process pretty painless.

By far, the easiest way to get hands working with objects is through the Object Manipulator behavior.

Simply add this behavior to your GameObject and configure settings in the editor. It supports one hand or two, and supports Move, Rotate, Scale, or all the above. Far Interactions are as simple as checking a box. And best of all, if your GameObject supports physics (see RigidBody), everything just works like you’d expect. Held objects will bump into other objects, and dropped ones fall with gravity! The link above should provide plenty of guidance, but note that NearInteractionGrabbable should be added to the object too. This is documented, just not called out clearly in the editor.

Finally, if you want to give visual clues about how an object can be interacted with, consider adding a BoundingBox. EnviroLight objects can’t be resized and are intended to resemble real-world objects as much as possible. Therefore, the decision was made to forego bounding boxes on EnviroLight objects.

Placing Objects

Many early HoloLens apps (including EnviroLight) used TapToPlace to position objects. The hologram would slide along the surface until the user performed an air tap. At that point, the object would stick in place. Unfortunately, this very handy script didn’t make it from HTK to MRTK. And though something similar could be built with SurfaceMagnetism, it was a little difficult to get right. Luckily, TapToPlace made a comeback in the MRTK with release 2.4 . This is a highly anticipated and highly welcome return!

In EnviroLight, each individual desk toy can be picked up with the hands. But the desk and all toys can be moved together as a group using TapToPlace.

Text to Speech

Most people don’t know this, but back in the day I wrote TextToSpeech for HoloToolkit. Sadly, this is one of the few components that aren’t available in MRTK because it doesn’t work cross-platform. This is unfortunate since EnviroLight uses it and it’s quite handy to have around. I do plan to look into creating a cross-platform version someday, but in the meantime it’s possible to use TextToSpeech.cs from the final release of HoloToolkit. This version still works great on HoloLens 2 and it doesn’t require any other dependencies.

Menus and UI

MRTK has incredible support for menus and user interfaces. In the early days, HoloLens apps used TagAlong to make UI elements follow the user. This same behavior can be accomplished with a RadialView, but apps now have many newer options to consider. If a 2D panel is still desired, consider starting with a Slate.

This floating window can be resized and pinned in space, or follow the user around.

If a menu is all that’s needed, consider using Near Menu:

Or even Hand Menu:

But probably the most important update to consider is how the hands will interact directly with UI elements. For this, MRTK provides a set of touchable controls that you can use to upgrade your classic Unity UI.

These controls work on HoloLens 1 and even touch screens too!

In EnviroLight, the primary method of interacting (other than voice commands) is an old computer terminal.

With this unique design in mind, a decision was made to stick with Unity UI but add support for Near Interactions. Obviously old terminals like this would never be touchscreen, but supporting Near Interactions feels more natural than requiring the user to always step back and use hand rays.

The process for adding Near Interactions to Unity UI is covered here, but the most important part is adding a NearInteractionTouchableUnityUI behavior and setting Events to Receive to Pointer. This causes the index finger to generate mouse pointer events for the UI, which ends up functioning very similar to a touchscreen!

MR Lighting Tools for HoloLens 2

The final milestone in this major update was to incorporate MR Lighting Tools for HoloLens 2. So you could imagine my surprise when I realized that it didn’t quite exist!

In June 2020, the MR Lighting Tools release on GitHub could be imported into a HoloLens 2 project without errors. But when the project was deployed to a HoloLens 2, errors would start appearing at runtime. These errors essentially meant that exposure, white balance and ISO weren’t getting set correctly. And without those, the images captured (and light generated) just weren’t right.

I reached out to the author and he confirmed that he hadn’t had time to work on issues yet related to HoloLens 2. But I was so close to finishing the project that I just didn’t want to wait! I ended up spending a weekend working on it and managed to submit a fully functional pull request on June 22nd. That PR was accepted and is now part of the master branch. Note though that as of June 27th a release build still needs to be produced, so if you need HoloLens 2 support you’ll need to download source.

Special thanks

First and foremost I want to thank Angel Muniz. Her skills helped create EnviroLight, and without her passion it likely never would have made it off the ground. If you’re ever in need of a talented artist that understands the performance needs of HoloLens, I highly recommend reaching out to Angel.

Next I want to thank Nick Klingensmith, the author of the amazing MR Lighting Tools. Thank you Nick for making the library, for teaching us how to use it, and for accepting my pull request so quickly! 🙂

Finally, I want to thank all of the incredible people in MRTK who are always open to feedback and endlessly strive to improve the library. A full list of everyone who helped me over the years would be too long to fit here, but for EnviroLight specifically I want to thank Bernadette Thalhammer, Cameron Micka, Catherine Diaz, Julia Schwarz and Will Wei.

Appendix

This last section is simply a listing of the GitHub issues and pull requests I created while working on EnviroLight. I’m including this for those who might be interested in seeing what it’s like working with open source software on a project like this. If you have your own projects in mind, this is a great community to be a part of!

#8051 – When a model with ObjectManipulator also has a RigidBody, there is a significant delay in responsiveness for near manipulations. I discussed the root cause as well as a possible solution.

#8052 – This is a pull request I submitted with a proposed solution to the physics issue above. The PR is currently being evaluated.

#8053 – TapToPlace.cs was overwriting the value of the SurfaceNormalOffset property on Start. This issue already has proposed fixes that will likely be accepted soon.

#8066 – When I turned on the option to render from the PV Camera, my holograms would not appear in Mixed Reality Capture. We determined it to be a Unity issue, but the team is planning to improve docs and inspector hints.

#8067 – SpatialObjectMeshObserver was running fine in the Unity editor but there were deserialization errors when deployed to the device. There is work in progress already to resolve this.

#8079 – I personally had some challenges in learning to use the Texture Combiner feature (for PBR slots in the new MRTK Standard Shader). I made recommendations on how the docs might be improved.

#8080 – Found that it’s not possible to call TapToPlace.StartPlacement before TapToPlace.Start has executed. There are scenarios where this may be necessary. There is currently an ongoing discussion.

#8081 – NearInteractionGrabbable only honors colliders on the same object where it is applied, but this is different than Rigidbody which honors all colliders found in children. This issue suggests supporting both.

#8082 – Initially I couldn’t get NearInteractionTouchableUnityUI to work. It turned out that I had to change Events to Receive to Pointer instead of Touch (the default). I recommended updates to the documentation.

#8083 – I found it unintuitive that NearInteractionGrabbable is also needed for ObjectManipulator but this is not enforced by script or inspector. I recommended at least a docs or inspector hint and possibly making it a hard dependency.