Final Evaluation
After we had implemented the changes, we went out and recorded 4 minutes of each of the 5 states and took it back to the lab to analyse. We quickly realised that our data looked strange. Instead of our data being separated into 5 distinct different states and in the order we recorded, there was a fair amount of muddle and cross over. We realised this may have been due to us accidentally clicking buttons and recording data when using the app. This meant that our data was inaccurate, resulting in us having to record again.
To avoid this happening again we implemented a few safety measures. Firstly, we got rid of the write button and made it so we automatically write to file every time we stopped recording. We also added a confirmation box that forced us to confirm the selected state before the app started recording. After this was done Sam, Celine and Oli went and re-recorded the data. After a final inspection, we were happy and ready to start prediction.
To avoid this happening again we implemented a few safety measures. Firstly, we got rid of the write button and made it so we automatically write to file every time we stopped recording. We also added a confirmation box that forced us to confirm the selected state before the app started recording. After this was done Sam, Celine and Oli went and re-recorded the data. After a final inspection, we were happy and ready to start prediction.
Prediction:
As explained in an earlier blog post, we devised a way that we thought would help us measure the effectiveness of each of our eight models. This is by using a scoring system in which each correct prediction (e.g. the model’s prediction matches our current state) we give them a point. We can then graph how many times each model gets a prediction right and for what state. The issue that arose with this is that after we changed how we trained the model we needed to update how we predicted. We did this by implementing a similar method to the recording activity that allows us to pick a predicting time and delay before predicting. This way we can ensure that we are performing the correct state when prediction starts allowing us to have cleaner data for graphing later.
After this was implemented by Oli and Sam, we had to perform the predictions. We decided that to give each state a fair chance of being correctly predicted, we would choose an equal time for each (one minute).
Results and Discussion:
Overall the Bayesian network (Bayes) models performed better than the K-nearest neighbour (IBK) models (Table 1). On average the Bayes models predicted the state correctly ~60% of the time while the IBK was correct ~50% of the time. The overall most effective data filtering method was the median (56-66%). Interestingly, the next best method was using the raw data (52-62%). The Mean method failed to improve prediction accuracy over using raw data and the root-mean-squared actually made the models worse (50-56%).
Table 1. Overall performance of state prediction models. As percentages.
After this was implemented by Oli and Sam, we had to perform the predictions. We decided that to give each state a fair chance of being correctly predicted, we would choose an equal time for each (one minute).
Results and Discussion:
Overall the Bayesian network (Bayes) models performed better than the K-nearest neighbour (IBK) models (Table 1). On average the Bayes models predicted the state correctly ~60% of the time while the IBK was correct ~50% of the time. The overall most effective data filtering method was the median (56-66%). Interestingly, the next best method was using the raw data (52-62%). The Mean method failed to improve prediction accuracy over using raw data and the root-mean-squared actually made the models worse (50-56%).
Table 1. Overall performance of state prediction models. As percentages.
IBK
Raw
|
IBK Mean
|
IBK Median
|
IBK
RMS
|
Bayes Raw
|
Bayes
Mean
|
Bayes Median
|
Bayes RMS
| |
Percentage correct
|
52%
|
51%
|
56%
|
50%
|
62%
|
62%
|
66%
|
56%
|
A closer look at the data:
We graphed the percentage of correct predictions for each of the states for the different state prediction models. There are some examples where IBK models were more effective than Bayes at predicting certain states with certain filtering methods (Figure 1). These included: Raw-running, Mean-running, Median-stairs, RMS-walking, and RMS-stairs. It is difficult to conclude that these models are actually more effective at predicting these states intelligently because the overall averages are low. This could imply that they are biased towards certain states which would increase the apparent accuracy of predicting those states but drag the overall average down. This would mean that the app is not intelligently determining the state but rather just choosing one state more than others.
Figure 1. State prediction performance with K-nearest neighbour trained models. As a percentage of correct predictions. Colours represent states.We graphed the percentage of correct predictions for each of the states for the different state prediction models. There are some examples where IBK models were more effective than Bayes at predicting certain states with certain filtering methods (Figure 1). These included: Raw-running, Mean-running, Median-stairs, RMS-walking, and RMS-stairs. It is difficult to conclude that these models are actually more effective at predicting these states intelligently because the overall averages are low. This could imply that they are biased towards certain states which would increase the apparent accuracy of predicting those states but drag the overall average down. This would mean that the app is not intelligently determining the state but rather just choosing one state more than others.
The Bayes models were very good at determining the sitting state with 94-97% correct predictions (Figure 2). This suggests that sitting is somehow distinct from the other four states in terms of the sensor readings logged during this action. This makes intuitive sense as the sitting state has the least physical movement and it also has the least chance of variability between training the prediction data sets. Notably, the Bayes-median model was also very good at predicting walking with 96% correct predictions. The ability of the Bayes-median model to predict walking and sitting is the main reason for its high overall accuracy. Note that it is less accurate predicting between skating, running and stairs (46-52% correct). The same can be said for all the other IBK and Bayes models except walking is also getting mixed up with these three states.
Figure 2. State prediction performance with Bayes network trained models. As a percentage of correct predictions. Colours represent states.
Conclusions and future work:
If our models were to pick the state at random they would be right ~20% of the time so given our models were correct over 50% we can conclude that our app functions at a basic level. That being said, even our best model (IBK-median) only predicted the state correctly 66% of the time on average which shows that improvements can be made.
The key to improving our apps overall functionality in the future is in getting it to distinguish between skating, running and stairs with more accuracy. The inaccuracy seen when predicting between these states could be caused by inconsistencies in form (how an action is performed) during training. We controlled for this by always having the phone in our pocket, in the same orientation. However, we trained the model using participants with different body types, gaits, skating styles (mongo/regular push), eagerness to get down the stairs and many more attributes. We also tested our predictions models using different participants which is a further source of confounding.
It should be noted that using multiple participants for training and testing predictions is not necessarily a bad thing. It essentially comes down to the intended purpose of the app. Do we want to make a personal fitness app that is trained by the user to know their movements then later give them feedback on them? Perhaps we want to make the same app but without the need to train it personally. Our method would be better for the latter with the caveat that we would need much more data from a much larger group of people.
It may be that three states skating, running and stairs are very similar in terms of the accelerometer and gyroscope readings logged while being performed. Intuitively, skating, running and taking the stairs are all higher intensity exercises with more jerky and random movements than walking or sitting. The easier and the least ambitious way to improve this would be to pick fewer and more distinguishable states but that is not very exciting. Another way to improve this would be to use the same person to train all states. Alternatively, we could increase the amount of data collected by a large number of people over all of the states.
We could also improve the prediction accuracy is by using cross-validation to find the best value for our sensor reading threshold (frequency of sensor readings). The threshold could be improved by making the app record sensor values for n-amount of different thresholds. This would allow us to train many different models simultaneously and we can see the threshold that performed the most accurately at testing time. We could then use that value for future predictions.
Another thing that could change is how we filter our data in terms of how batches of data are collected. Our batch size parameter is how many values are taken from the raw data before being sent to a transformation function. At the moment we take five values and transform them and send to file. We then take the next five values in the array and so on. This could be described as a folding window. We could switch to a sliding window which could give us better results as we would not be taking readings out of context when we created a batch. This would be done by taking the first five values and sending them away. Then the next set of values you send are taken by moving the batch up the array index by one. This could mean sudden jerks of the device may not be used out of context (e.g in different batches).
Our best version of the app (Bayes-median) can currently tell the difference between sitting, walking and some other activity (although we cannot tell which). This is promising for future projects if we decide to stick with this sensor and method of training. We have outlined some methods to improve this model with varying degrees of achievability which could get us closer to a fully functioning state predictor. The goal is to have this state prediction technology be accurate enough that it is not noticed, and is simply a function inside a separate app.
Credit: The work that went into this evaluation was performed by all members of the team.
Figure 2. State prediction performance with Bayes network trained models. As a percentage of correct predictions. Colours represent states.
Conclusions and future work:
If our models were to pick the state at random they would be right ~20% of the time so given our models were correct over 50% we can conclude that our app functions at a basic level. That being said, even our best model (IBK-median) only predicted the state correctly 66% of the time on average which shows that improvements can be made.
The key to improving our apps overall functionality in the future is in getting it to distinguish between skating, running and stairs with more accuracy. The inaccuracy seen when predicting between these states could be caused by inconsistencies in form (how an action is performed) during training. We controlled for this by always having the phone in our pocket, in the same orientation. However, we trained the model using participants with different body types, gaits, skating styles (mongo/regular push), eagerness to get down the stairs and many more attributes. We also tested our predictions models using different participants which is a further source of confounding.
It should be noted that using multiple participants for training and testing predictions is not necessarily a bad thing. It essentially comes down to the intended purpose of the app. Do we want to make a personal fitness app that is trained by the user to know their movements then later give them feedback on them? Perhaps we want to make the same app but without the need to train it personally. Our method would be better for the latter with the caveat that we would need much more data from a much larger group of people.
It may be that three states skating, running and stairs are very similar in terms of the accelerometer and gyroscope readings logged while being performed. Intuitively, skating, running and taking the stairs are all higher intensity exercises with more jerky and random movements than walking or sitting. The easier and the least ambitious way to improve this would be to pick fewer and more distinguishable states but that is not very exciting. Another way to improve this would be to use the same person to train all states. Alternatively, we could increase the amount of data collected by a large number of people over all of the states.
We could also improve the prediction accuracy is by using cross-validation to find the best value for our sensor reading threshold (frequency of sensor readings). The threshold could be improved by making the app record sensor values for n-amount of different thresholds. This would allow us to train many different models simultaneously and we can see the threshold that performed the most accurately at testing time. We could then use that value for future predictions.
Another thing that could change is how we filter our data in terms of how batches of data are collected. Our batch size parameter is how many values are taken from the raw data before being sent to a transformation function. At the moment we take five values and transform them and send to file. We then take the next five values in the array and so on. This could be described as a folding window. We could switch to a sliding window which could give us better results as we would not be taking readings out of context when we created a batch. This would be done by taking the first five values and sending them away. Then the next set of values you send are taken by moving the batch up the array index by one. This could mean sudden jerks of the device may not be used out of context (e.g in different batches).
Our best version of the app (Bayes-median) can currently tell the difference between sitting, walking and some other activity (although we cannot tell which). This is promising for future projects if we decide to stick with this sensor and method of training. We have outlined some methods to improve this model with varying degrees of achievability which could get us closer to a fully functioning state predictor. The goal is to have this state prediction technology be accurate enough that it is not noticed, and is simply a function inside a separate app.
Credit: The work that went into this evaluation was performed by all members of the team.
Comments
Post a Comment