Joe and Alex:
Recently we’ve been looking into using Unreal Engine (UE) to render and explore large scenes generated though our mapping solution. Whilst UE can handle these scenes with ease it’s also important for us to know that we can present them back to a user over a reliable web–based application.
Although there are several existing solutions for viewing and interacting with 3d renders of the world, one of the main limitations is the hardware requirements necessary to be able to run them at a reasonable framerate. This results in many popular devices (smartphones for example) being unable to fully utilize such software.
If you’ve found yourself in a similar situation then this guide will make a good companion in taking those first steps in presenting UE scenes on your front-end via pixel streaming. There are three main parts to the overall architecture to this solution, those being the UE engine application itself, the pre-existing UE node server, and a custom front end application. For our web-app framework we’ll be using Vue.js.
Within this tutorial, we focus on two main points:
- Sending data to the Unreal Application
- Receiving data from the Unreal Application
The Unreal Project
Before starting with this basic tutorial and setting up the Unreal application, we recommend you check out the official documentation for render streaming with Unreal Engine and familiarizing yourself with a basic setup outside of the iFrame environment. The documentation, as of version 4.24 can be found here (https://docs.unrealengine.com/en-US/Platforms/PixelStreaming/index.html).
The setup of the Unreal Project is identical to any other Pixel Streaming application set up in unreal engine, however to be able to better explain the necessary steps (and be able to replicate them); we will talk through an example set up. The example itself defines all of the blueprint logic within the level blueprint, though this can be moved to any actors blueprint as required.
Receiving a message from VueJS
Upon receiving a JSON payload from the VueJS application, we would like to be able to print the message within the payload to the screen. In order to do this, we can build a very basic blueprint to listen for events from the signalling server.
All messages sent to the pixel streaming application will be received by binding an event to the PixelStreamingInputComponent class method BindEventToOnInputEvent. To do this, we can bind a custom event to the BindEventToOnInputEvent node. The event descriptor will contain the JSON payload received from the VueJS application. By connecting the CustomEvent descriptor to a GetJsonStringValue node, we can then pull out the field we would like to print, in this case the “message” field.
By adding in a branch node and connecting to the success output of the GetJsonStringValue node, we can ensure the value was retrieved successfully before continuing. The final step is to connect the output string of the GetJsonStringValue node to the PrintString node as well as true execution of the branch node.
Although in our example we simply want to print the message field, you could handle custom cases by retrieving a type field from the JSON and handling each case accordingly.
Sending a message to VueJS
Within this send example, we would like to be able to click on an actor within the unreal scene and return the name of it to the VueJS application.
Although at first glance the send blueprint may look more complicated, it is actually pretty straight forward. Most of the nodes are used for retrieving the necessary unreal information and are only required if you wish to achieve the same goal.
The main node of interest is the SendPixelStreamingResponse node. The node needs to be connected to the instance of the PixelStreamingInputComponent class and in this example, will be executed when an actor is clicked. The SendPixelStreamingResponse takes in a string value, the contents of which are sent back to the VueJS application. Although in this example we are sending a string name, this can be any content that can be serialized to a string e.g. A JSON object.
Full Blueprint
Below you can see the full blueprint logic we used within the pixel streaming application, handling both send and receiving data.
The Node.js Server
The Node js server is using server-side rendering to serve a static html page to the address provided, with a full screen view of the scene currently being rendered in UE at the time. It uses webRTC and web sockets to provide a pixel stream and adds event listeners to the static html page to provide interaction data back to UE. The server runs be default on localhost port 80. Opening the directory and running “node server.js” will start the server and provide the page necessary for the vue application to render within the iFrame later.
A few small changes are necessary, first change the inputOptions object to use the controlSchemeType HoveringMouse, this makes the behavior smoother between the iFrame and the server. Next, to set up the interactions between the Vue app and UE add the two functions from the example GitHub repo, window.onmessage to add a listener for receiving messages, this then uses the emitUIInteraction function that is already defined to pass the payload on to UE. Add the handleUnrealMessage function in which uses window.top.postMessage to ‘forward’ messages sent from UE, now passing messages in both directions is handled.
The Vue.js Application
Our example Vue app is available from the GitHub repo, however if you’d prefer to set up your own vue app from scratch, follow these steps.
1 – Setting up the Vue App
To begin with you’ll need to install vue, using terminal command ‘$ npm install vue’, and create a new project ‘vue create pixel-streaming’, choose the options for router and history mode before changing directory into that project, finally ‘npm run serve’ will deploy a local verson of the vue application for development. Inside the views folder, create a new View to house the iFrame and pixel-streaming behavior. To show this view in the app include the router-link component in the main App.vue file, and use the index.js file within the router folder to make sure that a path to ‘/’ links to the new view you just created.
2- Setting up the iFrame
A simple template is all that is needed within the view, including an iframe with ID of “myIframe”. Here is where you must also include the location of the iFrame content, which we will bind to state using the ‘:’ shorthand. We use the signallingServerAdress of “ http://localhost:80 “ as this is where the node js server runs by default, this can be changed if running on different machines but works well for testing. We also included some simple CSS styling, but this is not necessary for a working project.
3 – Setting up the Streaming Logic and Receiving Messages
The Node.js server provided by UE will handle streaming, user interactions, and scene navigation out of the box, so by running that and pointing an iFrame from the vue app to the html page it serves will provide the same result but within a more useful front-end framework. The only caveat to that is you’ll need to handle some aspect ratio maintenance and propagate any custom events from the vue app through the server back to UE. Making use of the lifecycle hooks within Vue it is relatively simple to achieve this, define three methods and call these in the mounted hook; resizeIFrame, addResizeListener, and addListener. These methods will be executed as soon as the vue component has been created and mounted.
- resizeIFrame
- Changes the width and height style properties of the IFrame, using the iFrameScale of 0.75 (set in data) and the aspect ratio of 1.778.
- addResizeListener
- Adds an event listener to the window, on resize the resizeIFrame method will be called to maintain aspect ratio.
- addListener
- Add a listener for any cross-site communication, using window.onmessage. This method also includes logic to check the type of message is an object and to ignore webpack messages. Extra security can be put in place here to confirm the message is from the expected location. Currently on receiving a message a JavaScript alert is triggered displaying the message, but it is at this point the message can be used to trigger other methods or save data to state and is therefore where the Vue frontend can offer more flexibility than the node server itself.
4 – Setting up to Send Messages
One final method, sendMessage, is defined on the component. This method is used to send messages over to the node server. The example we provided uses a JSON message defined in data. This content is posted using .contentWindow.postMessage. This method is currently triggered by adding a click event to the button in our original template, the custom JSON message in data is also bound to the input from our original template as an example of what vue can offer in terms of reactivity when paired with the streaming provided by UE.
Get Started
If you’d like to take a look at the code and use that to get started head over to the GitHub page and feel free to post fixes or extra features. We’ve got more on the way soon.
https://github.com/Slingshot-Simulations/UnrealPixelStreamingExamples/tree/release