The goal for this project was to establish a communication channel between Unreal Engine and a C# WPF desktop application. The use case was that the desktop application would query the Unreal Engine application for data about the 3D world (e.g. the position or orientation of a certain asset).
Therefore, communication should be initiated from the WPF desktop application and should request data from the Unreal Engine application.
It was also a goal to be able to stream high frequency data at a later time. Finally, one point to consider which further complicates this is that Unreal Engine uses C++ while the WPF desktop application uses C#. Therefore, the chosen protocol must have an implementation in both languages.
I evaluated several ways how to establish said communication channel. Here are some options:
- Unreal Engine’s RPC calls
While this would be the best choice if both applications would be based on Unreal Engine, it’s not an option here. I couldn’t find an implementation of Unreal Engine’s RPC framework in C# and writing one on my own was out of scope. - MQTT
Supported as separate plugin maintained by Epic - good support. However, MQTT is best suited for sensor data and doesn’t support streams or bidirectional data out of the box. It also requires a separate broker application. Finally, MQTT isn’t optimized to transfer large amounts of data. - Websocket
Supported natively by the engine. Websockets are streaming by nature. However, websockets don’t define how data is to be serialized over the wire. Also, more implementation to support RPC calls and streaming different data types would have been needed. - HTTP calls
Supported natively. Here Unreal Engine would call a rest interface to fetch data. Just a Rest interface alone won’t support bidirectional data transfer or streaming data though. - gRPC
Not supported natively. However, various plugins are available (e.g. GitHub - TurboLink). gRPC supports unary RPC calls from client to server and single or bidirectional streaming. Additionally, it also comes with Protobuf which means serialization of data types and RPC commands are handled very well even for interoperation between different programming languages.
Based on the above analysis it became clear that gRPC would be my favourite choice. However, one problem remained - gRPC is optimized for client to server calls, not the opposite direction.
Unfortunately, there were no plugins which would implement a gRPC server in Unreal Engine. Therefore, I was stuck with building a gRPC server into the C# WPF desktop application and a gRPC client in the Unreal Engine application.
However, this meant that I had to find a way to allow the server (C# WPF desktop application) to initiate a communication and query the client (Unreal Engine application) for data (e.g. current position of an asset).
Like I explained above, there’s no way to start an RPC call from the server to the client in gRPC. However, there are different types of streams possible:
- Client to server streaming
- Server to client streaming
- Bidirectional streaming
The solution I decided to go for was to establish a long-running server to client stream so that the server can trigger the client and ask for an unary RPC call so the server can get the desired info (e.g. position data).
service Misc {
rpc CommandStream(CommandStreamRequest) returns (stream CommandStreamReply) {}
}
Often times though, a server might want to ask the client for all kinds of different data, so we need a way for the server (the C# WPF application) to request certain type of information. I figured that this is possible by adding a data structure for each command which can hold parameters that are transmitted from the server to the client. All the different command data structures are then added to the CommandStreamReply data structure using the oneof keyword.
service Misc {
rpc CommandStream(CommandStreamRequest) returns (stream CommandStreamReply) {}
}
message CommandStreamRequest {
}
message CommandStreamReply {
oneof command_parameters {
MoveCommandParameters move_command_parameters = 1;
InitializeCommandParameters initialize_command_parameters = 2;
GetPositionParameters get_position_parameters = 3;
}
}
Here’s what a typical exchange looks like if the C# WPF desktop application (gRPC server) needs to get an asset’s current 3D position from the Unreal Engine application (gRPC client):
Conclusion
I implemented this approach in a proof-of-concept project and it worked great. One catch is that we have to check that both server and client are configured to allow long-running streams, otherwise it will kill the stream after it has been idle for too long.