Compass Evaluation

Compass Testing

As well as the Estimotes we were given the task of evaluating the accuracy and precision of the compass on a cell phone. The phone’s compass app uses a combination of two sensors, the magnetometer (a sensor that measures the earth’s magnetic fields), and the accelerometer (which can tell where the cell phone is pointing) to come up with a digital equivalent of a traditional compass. The magnetometer determines where North and South are based on how the sensor interacts with the Earth’s magnetic fields and then takes the direction of the compass to calibrate that information and give us a result.

To test the accuracy and precision we first wanted to break down what is is to be “accurate” and what it is to be “precise”. Accuracy and precision may seem similar in a lot of way, but have two distinct meanings. We can say that something is accurate if it matches the ground truth; that is, it matches the desired outcome. For example, a dart thrower is accurate if they aim for the triple twenty and hit it. We can say that something is precise if all of its readings are similar to each other. For example, if a dart thrower was throwing darts and always hit the triple one then we would say that they are precise.

Once we broke down accuracy and precision we knew that we needed to find a “true” magnetic North; our equivalent of the triple twenty. To do this we are using Sam’s physical compass as our source of truth and we are going to compare the readings from the phone to this. We are making the assumption that Sam’s compass is 100% accurate for the sake of this test, even though we know that this likely isn’t the case. Unfortunately, we can’t afford and don’t have access to any extremely precise instruments, so a compass will suffice. We decided that for a fair test we should choose a location that will have a low amount of magnetic interference so that both the phone compass and the real compass will perform optimally. We picked Alhambra Rugby Park because it’s a large field with what we assume to be little interference and as an added bonus also in close proximity to University so will be easy for testing purposes.

Method:

1. Use the physical compass to establish where magnetic north is.
2. Place the phone facing North by lining up the top point of the phones compass with the pre-measured North measurement from our real compass
3. Wait until the compass settles on a number
4. Take the measurement
5. Exit the app so as to reset the magnetometer, and shake the phone around to reset the accelerator, this ensures the device is measuring fresh each time
6. Repeat this 5 times per set.

Results

	A	B	C	D	E	Average Error	Absolute Error
Test 1:	-6	-28	0	11	1	-4.4	9.2
Test 2:	7	11	6	1	-6	3.8	6.2
Test 3:	2	1	-1	-2	-3	-0.6	1.8
Test 4:	-3	1	-22	0	-2	-5.2	5.6
Total Average Error							-1.6
Total Absolute Average Error							5.7

Visualisations

Test 1

Test 2

Test 3

Test 4

Discussion

After conducting our tests we realized that using a compass as our anchor has several limitations. Further research showed that both mobile phones and handheld compasses use similar technology to calculate North (magnets). This meant that whichever interference was affecting one was likely affecting the other. We needed some other way of determining a true baseline (the measure which we say is accurate). We settled on using a sundial as our true north. Sundials are based on the sun's movements which research shows as being more accurate. To see the difference between what we had previously been treating as true North and the sundials alleged more accurate true North, we compared our compass to the sundial. This showed that our compass was about 18 degrees east of the sundials North. We converted this magnetic north to true north by adding Dunedin's magnetic declination (+25 degrees according to magnetic-declination.com). This resulted in a final calculation of -7 which means that our compass is seven degrees west of true north.





Although we conducted separate tests, the additional data only has limited use as we repeated the same method four times. At first while testing, there was quite a lot of variation between each reading, with some margins of errors as large as 28 degrees away from North. Interestingly, halfway through our testing (between test two and test three) we were prompted by the phone to re-calibrate the compass. This required “rolling a ball” around the screen. We looked into how and why the calibration process works. We found that the phone is using the rotational tilt to “sense the point where the phone axis is parallel to the North and South magnetic force line” and uses this information to work out which way North is. After calibration, we immediately saw an increase in both the accuracy and precision of the results, with the first five following readings only out by an average of 1.8 degrees.

We have two mean values, the absolute mean and the total mean. The total mean was -1.6 degrees. This means that on average the compass readings point 1.6 degrees West of the actual reading. The reason this is so low is because we take into account the negative values which are likely to average around zero (wisdom of the crowd theory). This value doesn’t mean much because a user isn’t going to take an average of a few values to get their reading. Consequently, using a wisdom of the crowd approach means we don’t mirror real life very well. The absolute mean was 5.7 degrees and indicates that each reading is on average 5.7 degrees off. This is more representative of what a user is going to get under typical use conditions. To put this number into perspective, we decided to use some basic trigonometry to find out how much this could affect a user. If you were to travel one kilometre with a compass 5.7 degrees off target, you would reach your destination 99m away. Obviously, as you travel further this number will only increase. In some situations, like mountain ranges, this would be unacceptable. For example, following the compass for an additional 99 kilometres would render you almost one kilometre off target; the distance line of sight between the Octagon and The Link.

Keep in mind, this average accounts for the outlier results of 22 and 28 degrees which happened both before and after recalibration, and is skewing our data significantly. We decided to include these values because although they don’t represent the average values we recorded, these are true readings that the compass recorded under the exact some conditions that a user might find themselves in. As such, it would be unfair to exclude them.

Conclusion

Our results show that our readings were reasonably accurate, ranging between -28 degrees and +11 degrees in the extreme cases away from North. We decided that for close range this would be acceptable however as distance increases, so will the error. This means that if travelling long distance the phone compass will not be accurate enough. However, we can say that the phone compass performs close to and as well as a normal hand held compass. Precision varied quite a lot with the phone compass. In some cases (immediately after calibration, see test 3) it performed extremely precisely, however in other cases there was a lot of variation between readings. If it was possible to calibrate the phone compass each use then we could be more confident in the compass performing precisely. Unfortunately, we are unaware of any ways to manually launch the compass calibrator and thus must make a judgement call on how the compass performed over the period of all tests. This leads us to believe that it is not very precise.