This week we got our test app working and then began our first test of the estimote beacons. We wanted to test how different the x and y coordinates were on the android app in comparison to the actual positions.
The phone was positioned 1.35 meters from the ground (the height of the beacons) using a tripod. We took measurements at meter intervals across the room, taking three measurements at each position. To take a measurement we opened our test app and waited for the reading to stabilise. We then closed and reopened the app in order to reset the app fully, to then take further measurements.
The difference between the sensor and actual x and y positions were calculated for each testing position by subtracting the actual value from the test value. Frequency plots were made to gauge the accuracy and precision of the sensors to locate the mobile device in the x- and y-axes separately
For the next analysis, we calculated the mean of the three measurements at each position and then calculated the difference from the actual position. Heatmaps of the x and y differences were plotted independently to see the accuracy of x and y position detection.
Pythagoras’ theorem was then used to work out the absolute distance between the test and actual values. These were then plotted as a heatmap in order to map out the overall accuracy of the sensor.
The histograms in Figure 1. have broad shapes which indicate that the beacons are not very precise for both the x- and y-axes. Only 21% of the x values are +/- 0.25 meters from the actual positions and 46% are between +/- 0.75 meters. For the y values 21% are +/- 0.25 meters and 40% are +/- 0.75 meters. The x values are distributed evenly around zero which indicates that with multiple measurements these beacons can detect your position accurately. It is important to note that this distribution is achieved with 90 measurements which is not how this technology is intended to be used. The y values are distributed less evenly and centred closer to 0.5. As for the x values, the number of y measurements plotted far exceeds a realistic use of the estimote beacons. Overall the beacons are not very precise and while they approach accuracy with multiple measurements, they are not very accurate with realistic use.
Figure 1. Histogram of Differences from Actual Position. Left X, Right Y.
The heat maps revealed some interesting biases and information about where the sensors are most accurate. We found that the accuracy of the sensors increases when the phone is positioned between the six sensors along the x-axis. This is visualised in Figure 2 (Top), where the outside positions show greater inaccuracy for x. Interestingly the accuracy decreases in a predictable way with positions on the left side showing positive differences and positions on the right side showing negative differences. This means the further away from the centre the room the mobile device is located along the x-axis the greater it underpredicts the distance from the centre of the room.
There is a similar effect seen for the y positions with more red squares in the bottom half (Figure 2 (Bottom). For the y measurements, the bias seems mostly toward the positive direction (top of the room) which was also seen in the skewed histogram in Figure 1. It is possible that some of the materials in the room are affecting the readings and creating this bias along the y-axis. Glass has a low interference potential for the Estimote beacons and is present on both sides of the room where the beacons are attached. The glass on the top side of the room has very prominent metal frames all the way along which is a material known to have a very high interference potential for the beacons. It would require further experiments to determine if this is the actual cause of the apparent bias, for example trying the same experiment but in a room that contains minimal metal and see if the same bias exists.
Figure 2. Heatmap Difference Between Sensor and Actual Position. Top X, Bottom Y.
The heatmap of absolute differences (Figure 3.) shows that positions near the centre and top of the room show the greatest accuracy. This data indicates that positions near the windows are the most accurate, however many of the Y positions for these were 5.55 which is the bounds of the room. We suspect if the position detected exceeds the bounds of the room then it defaults to the upper bounds. Because the upper bounds are only 0.55 meters away from the actual positions this has the effect of artificially lowering the absolute difference. Overall we believe the readings are best near the centre of the room between the sensors although there are some outlying points that challenge this notion.
Conclusion and Future Work
Overall our results show the beacons are not very accurate with realistic use and not very precise with a broad distribution of differences between the sensor and actual positions. The correct location +/- 0.75 meters is detected on the x or y-axes less than 50% of the time and sometimes it is completely wrong (3.5 meters off target for example). With the lack of precision in mind, a possible improvement to this system could be taking multiple readings before they are used for locating the position of the mobile device in the room.
We detected some biases in the readings with this set up with positions outside of the sensors along the axis. These readings are biased toward the centre of the room. It may be that our current setup with all the beacons on the top and bottom of the room is not the most optimal. We could try placing two beacons on the side walls and leaving two at the top and bottom and see how it compares. As previously mentioned we could also try this same experiment in a different room consisting of materials with less inference potential and see if accuracy and precision are increased.
Comments
Post a Comment