GPS on the Microsoft Hololens

Published January 18, 2017 by Damian Oslebo, posted by Damian Oslebo
Do you see issues with this article? Let us know.
Advertisement

Introduction

The Microsoft Hololens technology provides a myriad of possibilities, from its speech recognition to the gaze and gesture controls, all in an untethered environment. But there is one glaring omission, the lack of a GPS sensor onboard the device. This article provides a way to transmit a Bluetooth GPS signal from your smartphone (in this article, an Android OS) to the Hololens.

Explaining the Concept

This technique leverages Bluetooth LE advertiser/watcher technologies available on the Android and UWP platforms. Similar advertiser code is available on iOS, which has subsequently coined the use of the term "iBeacons" which speaks to all Bluetooth advertiser type devices. Unfortunately, this technology is not available on all devices and should be checked for during runtime. The benefit of the advertiser pattern on the smartphone is that a socket server (RFCOMM) connection does not need to be maintained on either the Hololens or Android side, reducing code complexity and making your application more crash proof. Another major benefit of the Beacon approach is lower overall power consumption on both the advertising and watching devices. I felt the easiest way to explain my technique was to divide the implementation into four sections, the first two are generic to all Bluetooth Beacon applications, and the next two sections describe the GPS specific implementation.

Implementation

BLE Advertiser on Android Java

Starting off with the content serving device, or Advertiser, the code below explains the initial setup of the data advertiser. GPS specific details are explained in other section. One thing to note the max size of an advertisement is 31 bytes. If the amount is exceeded, the advertisement callback will fail with a result of 1 or ADVERTISE_FAILED_DATA_TOO_LARGE. A check is highly recommended in both the application manifest and Java code to ensure the device has the permission and capability to produce Bluetooth LE advertisements. The user will be kicked out of the application to turn on Bluetooth if Bluetooth is off during activation. Add the following to your Android Manifest file: Add the following to your Android MainActivity file: public class MainActivity extends AppCompatActivity { BluetoothAdapter mBAdapter; BluetoothManager mBManager; BluetoothLeAdvertiser mBLEAdvertiser; static final int BEACON_ID = 1775; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); mBManager = (BluetoothManager)getSystemService(BLUETOOTH_SERVICE); mBAdapter = mBManager.getAdapter(); mBLEAdvertiser = mBAdapter.getBluetoothLeAdvertiser(); } @Override protected void onResume() { super.onResume(); if(mBAdapter == null || !mBAdapter.isEnabled()) { Intent enableBtIntent = new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE); startActivity(enableBtIntent); finish(); return; } if(!getPackageManager().hasSystemFeature(PackageManager.FEATURE_BLUETOOTH_LE)) { Toast.makeText(this,"No LE support on this device", Toast.LENGTH_SHORT).show(); finish(); return; } if(!mBAdapter.isMultipleAdvertisementSupported()) { Toast.makeText(this,"No advertising support on this device", Toast.LENGTH_SHORT).show(); finish(); return; } startAdvertising(); } @Override protected void onPause() { super.onPause(); stopAdvertising(); } private void startAdvertising() { if(mBLEAdvertiser == null) return; AdvertiseSettings settings = new AdvertiseSettings.Builder() .setAdvertiseMode(AdvertiseSettings.ADVERTISE_MODE_BALANCED) .setConnectable(false) .setTimeout(0) .setTxPowerLevel(AdvertiseSettings.ADVERTISE_TX_POWER_MEDIUM) .build(); AdvertiseData data = new AdvertiseData.Builder() .addManufacturerData(BEACON_ID,buildGPSPacket()) .build(); mBLEAdvertiser.startAdvertising(settings, data, mAdvertiseCallback); } private void stopAdvertising() { if(mBLEAdvertiser == null) return; mBLEAdvertiser.stopAdvertising(mAdvertiseCallback); } private void restartAdvertising() { stopAdvertising(); startAdvertising(); } private AdvertiseCallback mAdvertiseCallback = new AdvertiseCallback() { @Override public void onStartSuccess(AdvertiseSettings settingsInEffect) { super.onStartSuccess(settingsInEffect); String msg = "Service Running"; mHandler.sendMessage(Message.obtain(null,0,msg)); } @Override public void onStartFailure(int errorCode) { if(errorCode != ADVERTISE_FAILED_ALREADY_STARTED) { String msg = "Service failed to start: " + errorCode; mHandler.sendMessage(Message.obtain(null,0,msg)); } else { restartAdvertising(); } } }; private Handler mHandler = new Handler() { @Override public void handleMessage(Message msg) { super.handleMessage(msg); /* UI feedback to the user would go here. */ } }; private byte[] buildGPSPacket() { byte[] packet = new byte[24]; /* GPS code packet generation goes here */ return packet; }

BLE Watcher in UWP C#

The C# code for the Bluetooth advertisement is straight forward. The watcher listens for a manufacturer data id the same as in the Android application (in this example, 1775). As the watcher runs in another thread, you will need an application or event dispatcher to roll it back into the UWP / Unity application. You also need to add Bluetooth as a capability in the UWP Package Manifest. General UWP (Windows 10) code: public sealed partial class MainPage : Page { BluetoothLEAdvertisementWatcher watcher; public static ushort BEACON_ID = 1775; public MainPage() { this.InitializeComponent(); } private void Button_Click(object sender, RoutedEventArgs e) { if(watcher != null) watcher.Stop(); watcher = new BluetoothLEAdvertisementWatcher(); var manufacturerData = new BluetoothLEManufacturerData { CompanyId = BEACON_ID }; watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData); watcher.Received += Watcher_Received; watcher.Start(); } private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args) { ushort identifier = args.Advertisement.ManufacturerData.First().CompanyId; byte[] data = args.Advertisement.ManufacturerData.First().Data.ToArray(); var ignore = Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => { /* GPS Data Parsing / UI integration goes here */ } ); } } Adapted to ingest into Unity, do not forget to surround with #if NETFX_CORE preprocessor statements so the runtime editor does not complain. First, the role of the event dispatcher gameobject that acts like the Dispatcher in UWP for Unity (Also attach to a Game Manager object in the scene): using System; using System.Collections.Generic; using UnityEngine; public class EventProcessor : MonoBehaviour { public void QueueEvent(Action action) { lock (m_queueLock) { m_queuedEvents.Add(action); } } void Update() { MoveQueuedEventsToExecuting(); while (m_executingEvents.Count > 0) { Action e = m_executingEvents[0]; m_executingEvents.RemoveAt(0); e(); } } private void MoveQueuedEventsToExecuting() { lock (m_queueLock) { while (m_queuedEvents.Count > 0) { Action e = m_queuedEvents[0]; m_executingEvents.Add(e); m_queuedEvents.RemoveAt(0); } } } private System.Object m_queueLock = new System.Object(); private List m_queuedEvents = new List(); private List m_executingEvents = new List(); } Now the Unity GPS Watcher class: using UnityEngine; #if NETFX_CORE using Windows.Devices.Bluetooth.Advertisement; using System.Runtime.InteropServices.WindowsRuntime; #endif public class GPS_Receiver : MonoBehaviour { #if NETFX_CORE BluetoothLEAdvertisementWatcher watcher; public static ushort BEACON_ID = 1775; #endif private EventProcessor eventProcessor; void Awake() { eventProcessor = GameObject.FindObjectOfType(); #if NETFX_CORE watcher = new BluetoothLEAdvertisementWatcher(); var manufacturerData = new BluetoothLEManufacturerData { CompanyId = BEACON_ID }; watcher.AdvertisementFilter.Advertisement.ManufacturerData.Add(manufacturerData); watcher.Received += Watcher_Received; watcher.Start(); #endif } #if NETFX_CORE private async void Watcher_Received(BluetoothLEAdvertisementWatcher sender, BluetoothLEAdvertisementReceivedEventArgs args) { ushort identifier = args.Advertisement.ManufacturerData[0].CompanyId; byte[] data = args.Advertisement.ManufacturerData[0].Data.ToArray(); eventProcessor.QueueEvent(() => { /* Unity UI ingestion here }); } #endif } At this point, you can use an Android device to pass any information that you would like to the UWP application, but for the sake of this article, GPS data code is provided below.

GPS Sensor on Android Java

The hard part of setting up a Bluetooth advertiser/watcher is over, all that is left is to read the GPS sensor on your smartphone and serialize the information for an advertisement to the Hololens. Mentioned above, the advertisement will fail if the size of it is over 31 bytes. Luckily for me, I only required two doubles (8 bytes per double for the Latitude and Longitude) and two floats for my GPS data (4 bytes per float for the heading and speed data) for a total of 24 bytes. You also need to explicitly ask for permission to use location services in Android 6.0+ from the user, and it cannot just be in the application manifest. Add this to your Android manifest: Add this to your MainActivity file: LocationManager mLocationManager; static final int PERMISSION_RESULT_CODE = 1; Location currentLocation; @Override protected void onCreate(Bundle savedInstanceState) { /* * code from above section is same here */ int permissionCheck = ContextCompat.checkSelfPermission((Context)this, Manifest.permission.ACCESS_FINE_LOCATION); if (permissionCheck == PackageManager.PERMISSION_GRANTED) { startGPS(); } else { // No explanation needed, we can request the permission. ActivityCompat.requestPermissions(this, new String[]{Manifest.permission.ACCESS_FINE_LOCATION}, PERMISSION_RESULT_CODE); } } public void startGPS() { //Execute location service call if user has explicitly granted ACCESS_FINE_LOCATION.. mLocationManager = (LocationManager)getSystemService(Context.LOCATION_SERVICE); LocationListener listener = new LocationListener() { @Override public void onLocationChanged(Location location) { UpdatePosition(location); } @Override public void onStatusChanged(String provider, int status, Bundle extras) { } @Override public void onProviderEnabled(String provider) { } @Override public void onProviderDisabled(String provider) { } }; try { mLocationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0, listener); mLocationManager.requestLocationUpdates(LocationManager.NETWORK_PROVIDER, 0, 0, listener); currentLocation = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER); Log.i("GPS_Receiver", "startGPS: GPS Started.."); } catch(SecurityException e) { } } public void UpdatePosition(Location location) { currentLocation = location; restartAdvertising(); } /* Updated method */ private byte[] buildGPSPacket() { Location location = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER); byte[] packet = new byte[24]; if(location != null) { try { double latitude = location.getLatitude(); byte[] buffer = ByteBuffer.allocate(8).putDouble(latitude).array(); for (int i = 0, j =7; i < 8; i++, j--) packet = buffer[j]; double longitude = location.getLongitude(); buffer = ByteBuffer.allocate(8).putDouble(longitude).array(); for (int i = 8, j =7; i < 16; i++, j--) packet = buffer[j]; float bearing = 0; bearing = location.getBearing(); buffer = ByteBuffer.allocate(4).putFloat(bearing).array(); for (int i = 16, j =3; i < 20; i++, j--) packet = buffer[j]; float speed = 0; speed = location.getSpeed(); buffer = ByteBuffer.allocate(4).putFloat(speed).array(); for (int i = 20, j =3; i < 24; i++, j--) packet = buffer[j]; } catch (NumberFormatException e) { packet = new byte[24]; } } return packet; }

GPS Decoder in UWP C#

The last section of the decoder is the easiest since all that is required is deserialization of the data contained in the advertisement manufacturer data. I created a specific data class to handle the Data. using System; public class GPS_DataPacket { public double Latitude; public double Longitude; public float Heading; public float Speed; public static GPS_DataPacket ParseDataPacket(byte[] data) { GPS_DataPacket gps_Data = new GPS_DataPacket(); gps_Data.Latitude = BitConverter.ToDouble(data, 0); gps_Data.Longitude = BitConverter.ToDouble(data, 8); gps_Data.Heading = BitConverter.ToSingle(data, 16); gps_Data.Speed = BitConverter.ToSingle(data, 20); return gps_Data; } public override string ToString() { string lat, lng; if (Latitude > 0) { lat = string.Format("{0:0.00} ???????????????????????????????????'??N", Latitude); } else { lat = string.Format("{0:0.00} ???????????????????????????????????'??S", -Latitude); } if (Longitude > 0) { lng = string.Format("{0:0.00} ???????????????????????????????????'??E", Longitude); } else { lng = string.Format("{0:0.00} ???????????????????????????????????'??W", -Longitude); } return string.Format("Latitude: {0}, Longitude: {1}, Heading: {2:0.00}???????????????????????????????????'??, Speed: {3:0.00} knots", lat, lng, Heading, Speed); } }

Interesting Points

This technique provides the raw data to your GPS Android device; you still should follow some of the positioning filtering/business logic covered in the Android developer's guidance at: https://developer.android.com/guide/topics/location/strategies.html After you have started receiving the data you want to receive. Heading and Speed data can be unreliable if the user in not moving. Another gotcha is the explicit request permission calls that need to be made in the MainActivity file for GPS position access. I had added permission statements in the manifest but still had the application crash on security exceptions before adding this to the MainActivity file (new security feature of the Android 6.0+ I guess).

Conclusion

The Youtube video:
was used as a reference for the Bluetooth advertiser, it also has information on other Android Bluetooth LE techniques. I hope this article helps others with getting GPS position data into their Hololens (or any other UWP devices like tablets, XboxOne, etc.). I think this way of sending data is more robust (stable) than the GATT socket server alternatives to passing information across devices. I did not include the frontend code the GPS information in either the UWP or Android because I felt it was out of scope. Sample projects including this are available on request. Please feel free to comment or work on an iOS advertiser version that I can roll into the article later. Otherwise, thanks for the read and pass on if it is useful. And if I see this in the Unity store for anything other than free, I will be pretty upset.

Article Update Log

9 Sep 2016: Initial release 28 Mar 2017: Corrected typo in code for "buildGPSPacket()" under "GPS Sensor on Android Java"

Cancel Save
0 Likes 7 Comments

Comments

KCoyote123

cool

October 04, 2016 02:34 AM
bkfichter

Super helpful article.

Unless I am not seeing something, there is no definition for gps_receiver.getBestLocation() in the buildGPSPacket() method.

For the time being, I modified to

location = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER);

February 23, 2017 10:14 PM
jogvargas7

Hi!

Thank you for this article. I find it extremely helpful to what I am currently working on. Would you be able to provide the complete source code for sample projects that includes this implementation?

March 09, 2017 07:07 PM
Damian Oslebo

Super helpful article.

Unless I am not seeing something, there is no definition for gps_receiver.getBestLocation() in the buildGPSPacket() method.

For the time being, I modified to

location = mLocationManager.getLastKnownLocation(LocationManager.GPS_PROVIDER);

Thanks, original article corrected for change.

March 29, 2017 06:11 AM
olive11
Interesting. Still, as to 'hololens', what I'm wondering is how long will it be until everyone wearing the 'hololens' VR devices will be spammed everywhere with location-tracked advertising... its already there on the phone, but I can keep that in my pocket and ignore the ads. My prediction is that monetizing this interface will need to be done very delicately or these devices will be consigned to 'worn only when it can't be avoided' status. I can think of little that would be less appealing than to be walking about and have ads popping up in my vision as I walk past a store or look at a sign.
April 18, 2017 03:04 PM
olive11

Interesting. Still, as to 'hololens', what I'm wondering is how long will it be until everyone wearing the 'hololens' VR devices will be spammed everywhere with location-tracked advertising... its already there on the phone, but I can keep that in my pocket and ignore the ads. My prediction is that monetizing this interface will need to be done very delicately or these devices will be consigned to 'worn only when it can't be avoided' status. I can think of little that would be less appealing than to be walking about and have ads popping up in my vision as I walk past a store or look at a sign.

Still, giving it a second thought, I think augmented reality glasses have the potential to be interesting. My main concern is who controls the data that is generated on where and what I look at. The natural extension of AR is that information, advertisements, etc., materialize based on what the computer senses I'm looking at. If I'm in the Met looking at a Monet, an AR lens can display the Wikipedia article (or whatever) on that piece, which I can follow to Monet, which I can follow to Impressionism, etc. If I'm walking down the street I might see advertisements for businesses in my field of view.

But who controls the data on what I look at? In the abstract, things I spend more time looking at are things I find to be interesting. Will the computer controlling my lens display take note of the things I spend the most time looking at and report that data to servers who parse and sell my information, just like Google does with search queries? Will I have the option to opt out of such data collection? If the lens also has a microphone built in for voice commands, will it record my surroundings - including private conversations? Do I retain control of those recordings or will the EULA claim any and all things recorded by the device to be the property of the company?

Personally, I think the first use of any holodeck technology will be researchers in a lab demonstrating proof of concept. The second use will be the military for training. The third use will be the porn industry.
April 18, 2017 03:07 PM
Mark Hewitt

Many thanks for a great article. Here is my implementation based upon this: 

https://github.com/FlipWebApps/HololensGPS

October 15, 2017 07:54 AM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!

The Hololens is an amazing device capable of many things. Oddly enough, it is not equipped with a GPS Sensor. This article demonstrates a technique for pairing your GPS capable smart phone to the Hololens using Bluetooth LE signals.

Advertisement

Other Tutorials by Damian Oslebo

Advertisement