Experiments with a 3D browser
On March 30, 2016 Microsoft started shipping their first HoloLens as a development edition. Some time later, the MS Mixed reality platform was released, and distributed on Windows 10, along with the common updates. Suddenly we all had a 3D version of Paint, and a Mixed Reality Portal that allowed us to use VR glasses, without additional installation except the driver software for the device.
After I could persuade the company I work for to invest in a HoloLens, I started to think about the possible use for this. The obvious professional implementations for both HoloLens and VR are for people that work with 3D objects, like a car or other mechanical device that needs to be developed, furniture in house decoration or organs in a medical simulation.
But our company is mainly focused on information workers: how can one use this emerging technology in an office?
It is difficult to take a look in the future, but the most obvious thing that came to mind was that, like more and more applications are getting their web based equivalent in the cloud (Word, Excel etc.), the same might be valid for 3D developments. This means that, as we have for 2D, we need to be able to make and use 3D web sites and web applications.
Apart from some visual effects, the current Html specification does not support 3D features. I quickly stumbled upon X3D, a specification made for this (although not completely satisfying, as I will discuss later).
At the time, there was no 3D browsing possibility at all for HoloLens. So, I started to develop a browser that could run on HoloLens, Mixed reality devices, PC and Mac. Furthermore, I needed to develop some server components, so I could easily create 3D webpages with dynamic content.
Many more features where developed along the way. For example, one has to be able to adapt the content of a 3D page to the device used. The resolution of a HoloLens is much better than that of most VR devices. However, the field of view is much smaller (I am talking about HoloLens 1) and the overall performance is less compared to a VR display mounted to a modern PC with a VR enabled GPU card.
So on the HoloLens you can use smaller texts, but should skip a tag that shows a background skybox.
An experiment with dragging and rotation objects, using VR controllers. Objects that are dragged can be rotated in place (at a distance) by rotating the controller.
Being able to interact with each other on multiuser pages has its own merits and problems. I wanted people to use their avatar while going to a 3D web shop and start a conversation with the (avatar of) the owner. Or have an organization holding a meeting in a virtual office. Although the browser allows selecting one of the pre-defined avatars, it is sometimes preferable to have a look-alike avatar. There are currently shops that can scan a person and make a 3D model from it. After following a procedure for the preparation, the file containing the avatar can be stored anywhere on the web. When browsing any multiuser 3D pages the user starts with one of the chosen default avatars. But when his/her personal avatar is registered and the user is logged in, the browsers of other attendees to the website will immediately try to get and present the customized avatar.
Everybody on the multiuser website can activate or mute their microphone. This also means that the distance and location of the users should be considered, so anybody speaking from far away is not interfering with a local conversation (spatial sound).
Users should also be able to mute other users just for them, if they find somebody is annoying them.
For meetings and presentations, it is not sufficient to only share avatars and their voices. The presenter should be able to have a PowerPoint-like presentation. When selecting the next slide, the slide should switch in every browser connected for the meeting. Also, when user enters later in the scene of the presentation, the most recent slide should also become visible to him.
At some point somebody will want to show the screen of a desktop or show other ad-hoc content in the virtual environment. A separate component was developed for this. It allows dropping an image from file or clipboard, or a piece of text. The component uploads it to the server and presents it to the attendees of the presentation.
After I could persuade the company I work for to invest in a HoloLens, I started to think about the possible use for this. The obvious professional implementations for both HoloLens and VR are for people that work with 3D objects, like a car or other mechanical device that needs to be developed, furniture in house decoration or organs in a medical simulation.
But our company is mainly focused on information workers: how can one use this emerging technology in an office?
It is difficult to take a look in the future, but the most obvious thing that came to mind was that, like more and more applications are getting their web based equivalent in the cloud (Word, Excel etc.), the same might be valid for 3D developments. This means that, as we have for 2D, we need to be able to make and use 3D web sites and web applications.
Apart from some visual effects, the current Html specification does not support 3D features. I quickly stumbled upon X3D, a specification made for this (although not completely satisfying, as I will discuss later).
One of the earliest experiments with a simple web-shop-type 3D web on HoloLens.
At the time, there was no 3D browsing possibility at all for HoloLens. So, I started to develop a browser that could run on HoloLens, Mixed reality devices, PC and Mac. Furthermore, I needed to develop some server components, so I could easily create 3D webpages with dynamic content.
Simple and more complex basic-objects (prefabs) allowing to create quick loading 3D pages
Over time I came up with some features I wanted the browser to support:
- The 3D web pages should function in a similar way as their 2D counterpart, with more a or less HTML like structure with tags and attributes.
- Navigating should also be more or less the same, with hyperlinks leading to new content.
- Navigation within a web 'page', should be very consistent, having an easy implementation that allows 'walking' on surfaces, and a cursor indicating where these surfaces are.
- Have form support, with text fields, checkboxes etc.
- Support the use of more 3D models with textures, animations etc.
- Like with 2d websites, anybody should be able to have their own website on their own server.
- Allow to specify both single- and multiuser pages
- Allow interaction between users on the multiuser pages, including being able to start conversations. Users are presented by avatars
- Allow users to define their own avatar somewhere on the web and be able to use it on any multi user page.
Animated models created in Blender are used to show the possibility of having a more complex website, here inspired on Star Wars chess.
So on the HoloLens you can use smaller texts, but should skip a tag that shows a background skybox.
An experiment with dragging and rotation objects, using VR controllers. Objects that are dragged can be rotated in place (at a distance) by rotating the controller.
Another drag and drop example in a live-size chess board. Notice the 2nd visitor in the scene.
Two hands manipulation with HoloLens of an animated object.
Being able to interact with each other on multiuser pages has its own merits and problems. I wanted people to use their avatar while going to a 3D web shop and start a conversation with the (avatar of) the owner. Or have an organization holding a meeting in a virtual office. Although the browser allows selecting one of the pre-defined avatars, it is sometimes preferable to have a look-alike avatar. There are currently shops that can scan a person and make a 3D model from it. After following a procedure for the preparation, the file containing the avatar can be stored anywhere on the web. When browsing any multiuser 3D pages the user starts with one of the chosen default avatars. But when his/her personal avatar is registered and the user is logged in, the browsers of other attendees to the website will immediately try to get and present the customized avatar.
Users should also be able to mute other users just for them, if they find somebody is annoying them.
For meetings and presentations, it is not sufficient to only share avatars and their voices. The presenter should be able to have a PowerPoint-like presentation. When selecting the next slide, the slide should switch in every browser connected for the meeting. Also, when user enters later in the scene of the presentation, the most recent slide should also become visible to him.
Not yet as cool, but nevertheless a setup inspired by a scene from the movie 'Minority Report' with Tom Cruise
At some point somebody will want to show the screen of a desktop or show other ad-hoc content in the virtual environment. A separate component was developed for this. It allows dropping an image from file or clipboard, or a piece of text. The component uploads it to the server and presents it to the attendees of the presentation.
Reality portal in virtual space I: a web cam is filming me. This is presented on a screen in a virtual hall, in which my avatar is holding a presentation.
In the future it might be possible to mix users with avatar presentations with real people and objects. The closest thing currently possible is perhaps having real gathering, like a lecture or meeting, and add a large screen which shows the virtual room and attendees. at the same time, the virtual attendees see a large virtual screen showing the real event.
These films show me experimenting for such a 'reality portal' in virtual space.
Reality portal in virtual space II, now on HoloLens
Perspective: 3D will probably take its place in our daily applications, and will include 'extensions' like VR (immersive) glasses and augmented reality devices like the HoloLens.
X3D is a a somewhat older specification meant as 3D extension for browser technology. The initial setup however seems to have been to enable 3D parts in an otherwise 2D webpage. As I have discussed elsewhere the proper way to look at it is probably to take 2D content as a special case of a 3D virtual environment, making 2D an interactable 'texture' that we can apply on both flat- as well as on more complex surfaces. That way we can have traditional flat websites within the virtual environment, which can pop up 3d content when convenient.
v-Slam 3D browser: http://www.v-slam.org/
Source code: https://github.com/awonnink/vSlam
X3D is a a somewhat older specification meant as 3D extension for browser technology. The initial setup however seems to have been to enable 3D parts in an otherwise 2D webpage. As I have discussed elsewhere the proper way to look at it is probably to take 2D content as a special case of a 3D virtual environment, making 2D an interactable 'texture' that we can apply on both flat- as well as on more complex surfaces. That way we can have traditional flat websites within the virtual environment, which can pop up 3d content when convenient.
v-Slam 3D browser: http://www.v-slam.org/
Source code: https://github.com/awonnink/vSlam
Reacties
Een reactie posten