Points system added and round one testing conditions

Update:

Since our last blog we have updated the app in two ways. We now train models using both K-nearest neighbour and a bayes network. We also added a point system so we can evaluate and visualize the different training methods and filters.

New training method
The last version of the app only used the K-nearest neighbour method for training models. We decided as a team that it would be interesting to compare different learning algorithms to see which performed better and thus have an app with the greatest chance of correctly predicting your state. This meant that the app had to be updated to also train bayes network in parallel. This was relatively trivial because we could use the same data we recorded for K-NN as an input for the Bayes Net. Additionally the prediction part of the app had to be updated to show all eight, separate prediction methods (Figure below).

Prediction Screen. Displaying eight separate models and state selection for testing.

A points system for model evaluation

It was important that we knew how well each model performed in relation to each other so we can draw a conclusion about the best performing model(s). We displayed the prediction of a model on the app’s GUI which gave us an idea of how well models performed. This is far from a concrete measure however so we also introduced a points system. 

When the models made their prediction, we compared it to the actual state (assigned by the user) and gave them a score of '1' (correct) or '0' (incorrect) depending on whether they were correct or not. To do this, we adding a series of buttons to the prediction screen (Figure above) so the user can specify the current state while testing. For example, if we were walking we would hit the "Wa" button which would tell the prediction page that we were currently walking. 

A line was written to a file every time a sensor reading occurred while testing. Lines in the file would have the format: “0,0,0,1,1,0,0,1,WALKING”. The '1's and '0's indicate whether each of the eight predictions were correct or not and the "WALKING" tag specifies the actual state. This allows us to graph the prediction accuracy of each model and visualise it clearly (example below). 

Example of data and graphing. The test data will be processed something like this with bars 1,2,3, and 4 representing raw, mean, median, and RMS data respectively.

Capturing data
In terms of capturing our data, we implemented a few constraints in order to keep the training and test environments consistent. By keeping the environment consistent, it means we are able to train our model more accurately, therefore allowing our test model to present the correct state (more times than not). These constraints included the following:

1)  Holding the phone in our hand while performing the states - In order to get data within the same interval range for each state, we decided to train the data with the phone in the same position each time. For example, if you are running, we keep the phone in our hand while the arms are in running motion. Another reason we decided to have the phone in our hand instead of say our back pocket was due the way we implemented our application. The button for predicting and recording the data is on the screen and the tester needs to press the button. If it was in their back pocket, we wouldn’t be able to record the data. A drawback for this is that we won’t be able to test the state with our phone in our back pocket, which is where a phone is likely to be if we were running.

2) Recording data for each state for 2 minutes - Another way of keeping the data consistent to train our model is to test each state for 2 minutes. We did this by having one person record total recording time with a stopwatch, while the other person performed the state and told the application what state they were recording. This means that each model has equal amounts of data for each state, giving each state an equal chance of a correct prediction.

3) Keeping the tester the same for each state - In this case, we have used the same person to train the state. For example, Alex performed walking and running and Sam performed our walking down the stairs and skating states. When we go to test our model, we will use the same tester. This should hopefully maximise the performance of the models. This means we can also compare how well our model will work outside of a testing scenario (E.g with different users) by testing the model with someone who didn’t train the algorithm. For example, we can test running with Oliver instead of Alex to see if the model will still predict the state accurately.

4) Using the same data for both of the models (kNN (which is called IBK in Weka) and Bayes Network) - Similar to the reasons above, using the same data will allow us to compare the models fairly.

Written by James, Celine and Oli.