Finding the Best Model
Finding the best machine learning model
Data collection
We changed our training app so it outputs seven different types of data to be used for training the models. The app outputs mean, median and root mean squared (RMS) data in both sliding window and non-sliding window forms. It also outputs the raw data. Mean, median and RMS were calculated from five readings. The difference between the sliding and non-sliding window is that the non-sliding window keeps the batches separate so we end up with at least 80% fewer data points. With the sliding window version the raw data is shared so we end up with only 4 fewer readings than the raw data. There is about five times as much data for the amount of time spent training for sliding window although both methods are calculated from the same data ultimately.
Generating data and training models
To generate the training data we walked up and down the aisles in our test area for 20 minutes each to ensure that there was enough data to properly train the models. We then used this data to train five different models which were Bayes network, K-nearest neighbour, random forest, REP tree, and multilayer perceptron.
Testing models
The app was designed to test three separate models simultaneously so we tested Bayes network, K-nearest neighbour, and random forest first. Our app allows us to choose which aisle we are actually in and then it automatically scores the predictions for us and outputs them as .csv file to be graphed. We tested each aisle for two minutes before extracting the results and graphing them. We tested multilayer perceptron and REP tree separately which was not ideal but we tested it in the exact same way and they both used the same training data.
We changed our training app so it outputs seven different types of data to be used for training the models. The app outputs mean, median and root mean squared (RMS) data in both sliding window and non-sliding window forms. It also outputs the raw data. Mean, median and RMS were calculated from five readings. The difference between the sliding and non-sliding window is that the non-sliding window keeps the batches separate so we end up with at least 80% fewer data points. With the sliding window version the raw data is shared so we end up with only 4 fewer readings than the raw data. There is about five times as much data for the amount of time spent training for sliding window although both methods are calculated from the same data ultimately.
Generating data and training models
To generate the training data we walked up and down the aisles in our test area for 20 minutes each to ensure that there was enough data to properly train the models. We then used this data to train five different models which were Bayes network, K-nearest neighbour, random forest, REP tree, and multilayer perceptron.
Testing models
The app was designed to test three separate models simultaneously so we tested Bayes network, K-nearest neighbour, and random forest first. Our app allows us to choose which aisle we are actually in and then it automatically scores the predictions for us and outputs them as .csv file to be graphed. We tested each aisle for two minutes before extracting the results and graphing them. We tested multilayer perceptron and REP tree separately which was not ideal but we tested it in the exact same way and they both used the same training data.
We tested our models using raw data instead of matching the test data type to training data type. We made this compromise so all the differently trained models could be tested with the same data. This also gave us the advantage of having more data to test our models for the amount of time spent testing. There was also a problem with whether we had collected enough data to calculate means, medians and RMS for testing in a useful way because in practice we only receive about one reading from the estimotes per second. With our setup, five readings are required for calculations as you could easily be in five completely different positions within this period.
Results
Overall raw, median and mean data generated the best models. The models trained with RMS data would only ever predict one aisle. This suggests that RMS is a bad fit for our data although this could be a result of a programmatic error. As RMS is universally bad we will not discuss it further.
Figure 1. Accuracy of Bayes Network
Bayes network (Figure 1) performed best with median and raw data with both having 92-93% accuracy overall. The sliding window did not make a significant difference to our model’s overall accuracies although it can be seen that it reduces bias towards aisle two for the mean.
Figure 2. Accuracy of K-nearest neighbour
K-nearest neighbour (Figure 2) performed best with mean data with 90% overall accuracy. The raw or median data achieved 82% overall accuracy. The sliding window actually made the models worse in this case with the mean data showing 87% for sliding window compared to 90% for non-sliding.
Figure 3. Accuracy of Random Forest
Random forest (Figure 3) also worked best with mean data (91%) and was overall better with non-sliding window. We had originally trained the models with 10 minutes training per aisle and tested with Bayes network, K-nearest neighbour and random forest. Under these conditions random forest median with sliding window was one of the best options with about 80% overall accuracy however with more training it was overtaken by the other methods. This goes to show the importance of the length of time spent training on the accuracy of the models.
Figure 4. Accuracy of REP Tree
REP tree performed very poorly under all conditions with it mostly predicting the same aisle every time (Figure 4.)
Figure 5. Accuracy of Multilayer Perceptron
Multilayer perceptron was by far the best model (Figure 5). It performed with 98% or greater overall accuracy in all cases except RMS. Raw, non-sliding mean, and non-sliding median were essentially 100% accurate, only making one wrong prediction in 257 total predictions.
Discussion
Raw performed better than expected with the estimote data. This is most likely because our training and test data matched. In assignment two we found median was the best option for preprocessing our data but that data was very different to this data. The gyroscope and accelerometers data was prone to outliers and it was collected at a higher frequency. The estimote data may be less prone to outliers and the frequency of collection is much lower. We set our app to collect data every 100 ms but in reality, the signal strength was only updated about once per second. As a result, we found the data did not jump around as much which is another reason why raw data could be performing well.
It was interesting that the sliding window did not generally perform as well as the non-sliding window data. It is difficult to tell why this is the case with this small data set but if we had to speculate, it might be caused by a mismatch between the training data and testing data. The training data is in sliding window form and the test data is raw which may have thrown the predictions off in a way that didn’t affect non-sliding window as significantly.
For the final app, raw data with the multilayer perceptron model is our best option because of its nearly 100% accuracy under our test conditions. The reason for picking raw over median or mean is due to our ability to collect raw data at a higher frequency allowing our app to check the user’s position about once every second instead of every five seconds.
The code was written by Oliver and Alex and report was written by James and proofed by Celine and Sam.
Results
Overall raw, median and mean data generated the best models. The models trained with RMS data would only ever predict one aisle. This suggests that RMS is a bad fit for our data although this could be a result of a programmatic error. As RMS is universally bad we will not discuss it further.
Figure 1. Accuracy of Bayes Network
Bayes network (Figure 1) performed best with median and raw data with both having 92-93% accuracy overall. The sliding window did not make a significant difference to our model’s overall accuracies although it can be seen that it reduces bias towards aisle two for the mean.
Figure 2. Accuracy of K-nearest neighbour
K-nearest neighbour (Figure 2) performed best with mean data with 90% overall accuracy. The raw or median data achieved 82% overall accuracy. The sliding window actually made the models worse in this case with the mean data showing 87% for sliding window compared to 90% for non-sliding.
Figure 3. Accuracy of Random Forest
Random forest (Figure 3) also worked best with mean data (91%) and was overall better with non-sliding window. We had originally trained the models with 10 minutes training per aisle and tested with Bayes network, K-nearest neighbour and random forest. Under these conditions random forest median with sliding window was one of the best options with about 80% overall accuracy however with more training it was overtaken by the other methods. This goes to show the importance of the length of time spent training on the accuracy of the models.
Figure 4. Accuracy of REP Tree
REP tree performed very poorly under all conditions with it mostly predicting the same aisle every time (Figure 4.)
Figure 5. Accuracy of Multilayer Perceptron
Multilayer perceptron was by far the best model (Figure 5). It performed with 98% or greater overall accuracy in all cases except RMS. Raw, non-sliding mean, and non-sliding median were essentially 100% accurate, only making one wrong prediction in 257 total predictions.
Discussion
Raw performed better than expected with the estimote data. This is most likely because our training and test data matched. In assignment two we found median was the best option for preprocessing our data but that data was very different to this data. The gyroscope and accelerometers data was prone to outliers and it was collected at a higher frequency. The estimote data may be less prone to outliers and the frequency of collection is much lower. We set our app to collect data every 100 ms but in reality, the signal strength was only updated about once per second. As a result, we found the data did not jump around as much which is another reason why raw data could be performing well.
It was interesting that the sliding window did not generally perform as well as the non-sliding window data. It is difficult to tell why this is the case with this small data set but if we had to speculate, it might be caused by a mismatch between the training data and testing data. The training data is in sliding window form and the test data is raw which may have thrown the predictions off in a way that didn’t affect non-sliding window as significantly.
For the final app, raw data with the multilayer perceptron model is our best option because of its nearly 100% accuracy under our test conditions. The reason for picking raw over median or mean is due to our ability to collect raw data at a higher frequency allowing our app to check the user’s position about once every second instead of every five seconds.
The code was written by Oliver and Alex and report was written by James and proofed by Celine and Sam.
Comments
Post a Comment