Home Straight

Update - Bad Predictions:

When it came to recording the states, we chose to keep the recording time across each state the same (two minutes) for consistency purposes. However, we then found that recording someone falling for two minutes was not very practical. This was because it took less than a second to “fall”, making it difficult for us to tell the app exactly when the fall started and stopped. Because of this, we decided to make a last-minute state change from falling to skateboarding. This didn’t require much change to the code as it was mainly just changing variable names and UI elements. 

After we made this change, Sam and Al went home and recorded the skateboarding state for two minutes. With this done, we could train the models and start making some predictions. What we noticed straight away was that all the models were predicting very inaccurately. After realising all the models were performing poorly, the team sat down and had a much-needed whiteboard session. During this, we discussed all the possible factors that could be affecting our readings negatively. From this session, we came up with several potential causes, and a potential solution to mitigate these issues to help improve our models.

Causes:

1. Not enough data

At the moment we are recording two minutes worth of data per state. It’s a fair assumption to say that increasing the amount of data we have will help the model to predict better. Therefore, we decided that we will increase the data recording time from 2 minutes to 4 minutes per state. 

2. Holding phone

One of the main issues affecting our results may be due to holding the phone in our hand and adding unnecessary acceleration/orientation to it while we record instead of having it in our pocket. For example, if we tested the ‘sitting’ state on the table, there are no other constants that will affect the gyroscope and the acceleration. Essentially only 1g affecting the phone. What is generally affecting the readings is the “cradle effect” that is created by holding the device out front. Basically, our arm is adjusting to keep level for when the device should be shaking around. This is one of the variables that we have decided to change and will do so by recording data in the pocket, this will require a small code restructure which we will discuss further down. 

3. Formulae

Once the code was written to apply filters (such as median, mean and RMS), we didn’t double check whether the values we were getting were actually correct. Therefore, we thought if we were calculating these filters incorrectly, then it could be causing the bias. We ended up ruling this option out early on as Alex double checked the values in excel to see if everything matched up (which it did). 

4. Accelerometer not working

One of our guesses was that the accelerometer was not performing properly. When Sam was skating, the model was predicting every single state (even sitting). This was a worrying sign that led us to believe the accelerometer was not operating properly. We tested this by checking if the values we were recording matched the expected output of accelerometer data, and by training only 2 states. These 2 states both consisted of sliding the phone across the table (one state was really slow and the other much faster). In order to keep the gyroscope from affecting the final data, the phone was kept as straight and as flat as possible while sliding the phone across the table. After recording and training, both states were predicted with high accuracy which then meant we could rule out the accelerometer

5. Threshold

The threshold is what we use to check how close a gyroscope and accelerometer reading has to be to one another for us to consider them occurring “at the same time”. We are thinking our current threshold of 100ms is too large and could be causing an issue with the relation of gyroscope and accelerometer within each data point. We have decided to drop this variable, and as we don't have the time to train data for each threshold, so we can only make an assumption on how low to take it. If we had more time we would use cross-validation to find the best value for this.

6. Batch size

The batch size is about how many readings we use to take an average, median etcetera. This is one of our tuning parameters, however, after discussion, we decided that we didn’t think this would be likely to be causing a large effect on the prediction. Potentially this parameter could be changed to act as a sliding window instead of a skipping one, this way we don’t pull data out of context when we compute an average. We have decided to leave this variable as it is for now, and potentially change it later on if the model is still underperforming.

Future Code Changes

Now that we want to record data inside the pocket instead of in the hand, our app requires a small rework. The initial concept was simple; to add a delay timer to our recording button, and a countdown timer for when to programmatically stop recording the state. The delay function is so that we can take the phone and put it inside your pocket before the user starts recording, and the countdown timer is how long we want to record the state for before the recorder stops. This countdown timer was necessary, as pulling the phone from the pocket to stop the recording would corrupt the gyro data. This rose the argument as to what we code each of these variables to for each state initially. It is easy to do 4 minutes of walking in a straight line but 4 minutes of stairs without stopping is near impossible. So we came up with the idea to use text input UI elements that we can input delay and recording time into. This way we can customize our delay and recording time, this is especially helpful so that we don't have to update the code every time we want to train for a different state. Due to the way our points scoring system works, we will have to do the same for the prediction page, doing this means our points system won’t score all zeros whilst we wait for the user to pocket the device and start performing the state. The recording time input for state prediction allows us to predict each state for equal amounts of time and it stops scoring automatically. This means when we go to produce our graphs, all data captured will be represented fairly.

Once all this has been completed, we will again go out and record our training data. After we have recordings for all of the states, we can go out and perform our prediction testing. We plan to test how well our models predict by starting the predictor and performing the state for a set amount of time. 

Written by Sam, Alex, and Celine