Hardware components | ||||||
 |
| × | 1 | |||
| × | 1 | ||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
Hand tools and fabrication machines | ||||||
 |
| |||||
Smart shirts exists, but they can be very expensive. Most are $300+, which can be very high for a new gym goer. These expenisve shirts are made for professional athletes or researchers who have lots of funding, not for the general public. The shirt presented in the project could be used for many applications including:
- Personal Use - for the average user to understand more about their workouts and progress.
- Online Personal Training - For trainers to manage their clients over vast distances. The trainer would have access to the clients data and be able to give their own feedback on what their clients need help with.
- Better Organization of Fitness Classes - organizations can collect data on how clients respond to their programs throughout a given class. Seeing where clients experience difficulties (where heart rate increases to fast, improper muscle activation, breathing rate not consistent, etc) would allow the organizers of the class to adjust the program in order to receive better response from their clients.
- Research - Using sensors to collect desired exercise data from the participants. Researchers could use this very important data to do their analysis.
The smart shirt, in this project, is for low budget consumers who are not very experienced in the gym or seek a device that can correct their form, and give them feedback on their performance. The purpose for this project was to create a proof of concept that such a shirt could be created and it was successful. Where infrared sensors could be used to detect relative strength of muscle contraction (to distinguish between a good contraction and a weak contraction) to replace the use of an expensive contraction sensor (such as the MyoWare muscle sensor). One muscle contraction sensor usually costs $50+ and one infrared sensor costs under $1. When there is 5-6 sensors on a shirt, the difference in price becomes very clear. To strengthen the use of IR sensors, a supervised machine learning algorithm was introduced to the logic flow of determining a good form repetition (or "rep"). The algorithm was successfully trained to distinguish between a good rep, a bad form rep (leaning forward or bending back are common bad form indicators) and a weak contraction also known as a "half rep".
For those that do not know what supervised machine learning is, I suggest a thorough read through of this. But basically a programmer gives a ton of data to an algorithm, where each piece of data is a set of vectors and some kind of classifier. For example, if a programmer wanted to distinguish between an apple and a banana in an image, the programmer would have a set of vectors representing shape, colour, texture and then give that set a name (the classifier). After processing all the data (known as training the algorithm) the machine learning algorithm will create a function to tell the difference between apples and bananas.
In this tutorial I will attempt to describe how the project was completed while trying to remain high level. The instructions are still given and future blogs will be posted to explain each aspect better. But for now, it is in everyone's best interest to actually understand why certain procedures/precautions were taken, before attempting the tasks at hand. Moreover, this allows anyone trying to recreate this app to use different platforms but still use the same logic. Lets get started!
Top Level View of Project and The Logic FlowThis section describes in full the flow of how the app works. From how the data is collected to how the user receives the exercise information. The information in this section will give a good overview of what needs to be done to tackle a project like this. Then the following sections will go into more detail on each procedure.
- Step 1, User Connects: The first thing to do is obviously connect the Bluetooth devices to the phone application. This is done through Bluetooth low energy. Although I would prefer to use Bluetooth serial for an application like this, both micro controllers use Bluetooth low energy. The program is made in the portable class of Xamarin and can be used on Android and Apple products.
- Step 2, User Starts Set: The second thing is to obtain information from the user of the shirt once the user has started a set. This is done through the use of the ST SensorTile and the LightBlue Bean. Information for acceleration is acquired using the on-board accelerometer in the micro-controllers. Then the muscle contraction data is acquired using the infrared sensors. This data is packaged in a Scratch Characteristic and sent to the phone application every 100ms where it is collected into buffers for later use.
- Step 3, User Ends Set: When the user ends the set the post processing component begins. An algorithm is used to detect the reps completed during a set. Each rep is then passed to the machine learning algorithm for analysis.
- Step 4, User Feedback: Once the machine learning algorithm is done its analysis, the user receives a list of each rep and how they performed. Right now the application is trained for three possible scenarios, good form rep, bad form rep, and a weak contraction rep (also known as a "half rep").
This concept was based off the same technology that popular companies like Fitbit and Apple use to retrieve heart rate monitor data from the user. It should first be noted that infrared sensors are reflective sensors. An LED transmits a light and then the photo diode receives the light and returns a reading based on how much of the transmitted light was reflected back. In this case, the transmitted light reflects off the blood in the area under the sensor (therefore try to pick good places to put the sensors such as thin skin or on veins). The intensity of the light changes based on the volume of blood under the sensor. This allows heart rate detection because when the heart pumps the volume of blood increases and the infrared receiver gets peaks at those moments. The micro-controller then calculates the heart rate based on the time spacing between the peaks.
For muscle contraction sensing, the interest is still volume of blood. When a muscle is contracted more blood flows in the area of the muscle. Thus, peaks can be detected in the same way they are in heart rate monitor sensors. The figure below displays a user contracting their muscle for 3 reps. The data was filtered with a low pass filter with a cutoff of 1 Hz. This is because a contraction during a workout does not go faster than 1Hz, and this way all the noise in the system is rid of.
Notice how there are 3 peaks representing when the user had their muscle most contracted. One of the biggest problems faced is the fluctuation of where a rep starts and ends. In other words, a peak does not always happen at 890 and a relaxed muscle is not always start at 870. The fact the volume of blood remains higher in that area once a muscle is contracted (this contributes to what gym goers call "The Pump"). Another factor is heart rate increases during exercise and therefore, the blood volume increases as well. This made it very hard to create a logic flow where a value of X would mean a good contraction and a value of Y meant the muscle is relaxed and therefore the rep is over. Hence, local maxima and minima algorithms were used to find these peaks and valleys.
Local Maxima and Minima Algorithm For Detecting RepsThe name is very self explanatory, the algorithm attempts to find the maximum and minimum within a certain frame of data. The algorithm checks each point and determines if it is the current maximum of the data set, if not the algorithm moves on to the next point. However, if the point is deemed the kind of the hill (get it, its at the top, because its the peak point!) then it is stored for later. The same process is done for finding the local minimum values. Since a repetition should never take less than 3 seconds, I made the window size 3 seconds long. Since the software samples at 100ms (or 10 points/ second) the window size is 30 data points long. Once the algorithm is done there is two lists containing the the local maxima and minima. The maxima in this case would refer to the peak contraction and the minima would refer to when the muscle is relaxed. During an exercise the muscle continuously switches between being contracted and relaxed. Therefore, a proper repetition would be from when a muscle is initially relaxed followed by a contraction followed by the muscle being relaxed again. Every time this sequence happens in the post processed data, the rep data is extracted to be sent to the machine learning algorithm, likewise, a rep is counted so the user does not have to input that information. Each rep data is normalized to be comparable to other rep data for the machine learning algorithm. The code sample below demonstrates a method to find the local maximas in C#:
static List<int> FindMaxPeaks(List<double> values, int rangeOfPeaks)
{
List<int> peak_points = new List<int>();
int checksOnEachSide = rangeOfPeaks / 2;
for (int i = 0; i < values.Count; i++)
{
double current = values[i];
IEnumerable<double> range = values;
if (i > checksOnEachSide)
{
range = range.Skip(i - checksOnEachSide);
}
range = range.Take(rangeOfPeaks);
if (current == range.Max())
{
Boolean found_flag = true;
for (int j = 0; j < peak_points.Count; j++)
{
if (found_flag && (i - peak_points[j]) < checksOnEachSide)
{
found_flag = false;
}
}
if (found_flag)
{
peak_points.Add(i);
}
}
}
return peak_points;
}
Machine learning is a buzzword thrown around a lot that people misunderstand for some sci-fi magic that will some day make robots our overlords. But it is really just a bunch of statistics that is quite simple to use. The most important part of a machine learning algorithm is the DATA. The programmer must supply the algorithm with the right data and enough data to get the job done. Earlier I explained an example about a machine learning algorithm to know the difference between an apple and a banana. The right data (as explained before) would be things such as colour, shape, or texture. These characteristics (or vectors) give humans clear distinction between the two. However, imagine one of those characteristics the type of food it is? Well, they are both fruit. If someone went to Booster Juice and asked "I want fruit in my smoothie", the employee would not be able to distinguish between giving that customer a banana or an apple. The programmer must figure out ways to separate the classifiers through the vectors. This is where machine learning gets interesting. The hardest part of the machine learning process is the pre-processing before the data is fed to the algorithm, especially with time domain data. The apple/banana algorithm has no time domain component as all vectors relate to physical entities associated with a single value. However, a whole chunk of time domain data cannot be fed into the machine learning algorithm all at once. Statistical functions, such as max, mean, variance and standard deviation, must be used on each chunk of data before being pushed to the machine learning algorithm. Proper form is very important for exercising safely. One of the biggest differences between good form and bad form is body alignment. If a user has good form during a bicep curl, there torso should stay upright and never lean forward or backward through out the exercise. Through inspection of the vectors used in this project, one would notice that many are based off the three axis acceleration. The mean, max, standard deviation and variance are all fed to the machine learning algorithm to detect good form. In theory, if the form was good during the exercise, the mean and max should be very close, and the standard deviation should be smaller than that of bad form. Through inspection of the vectors used in this project, one would also notice mean was not used for the muscle contraction data. This is because the mean data would not yield significant results as the mean would vary significantly. To determine a strong contraction versus a weak contraction, rep length in seconds, the max value and standard deviation is used from the rep data.
Once the programmer knows the data they need, they must collect a lot of it. Many repetitions of each scenario were performed and logged for the machine learning algorithm. The algorithm was then trained and tested with this data, giving substantial results. The algorithm used was a Multiclass Decision Forest with 8 decision trees. Many algorithms were tested against, this had the best results. The figure below shows the results from the testing phase.
From the training data 70% is used to train the model and then 30% is used to test how accurate the model is. During the training phase the model was proven to be 92% accurate, with only the half rep classifier being below 80%. This is because the half rep IS also bad form, therefore, a lot of the characteristics end up being the same. Likewise, this model was trained with under 100 points, this could be easily solved with collecting more data about half reps and bad form. Although, in this case, the user getting the feedback for bad form when they are doing half reps is still significant enough for them to change their approach the exercise.
Creating a Microsoft Azure Machine Learning Web Service can be done by following this well documented tutorial on the subject. The figure below shows the flow chart created in Microsoft Azure Machine Learning Studio.
Below is the code that shows how to send data to be predicted by the machine learning algorithm. The code below also shows all the values used for training the machine learning algorithm, where the characteristic "class" is the predicted value.
public async Task InvokeRequestResponseService()
{
using (var client = new HttpClient())
{
var scoreRequest = new
{
Inputs = new Dictionary<string, List<Dictionary<string, string>>>() {
{
"input1",
new List<Dictionary<string, string>>(){new Dictionary<string, string>(){
{
"RepLegth", RepDataForMLS[0]
},
{
"XAcc_max", RepDataForMLS[1]
},
{
"XAcc_mean", RepDataForMLS[2]
},
{
"XAcc_var", RepDataForMLS[3]
},
{
"XAxx_sd", RepDataForMLS[4]
},
{
"YAcc_max", RepDataForMLS[5]
},
{
"YAcc_mean", RepDataForMLS[6]
},
{
"YAcc_var", RepDataForMLS[7]
},
{
"YAxx_sd", RepDataForMLS[8]
},
{
"ZAcc_max", RepDataForMLS[9]
},
{
"ZAcc_mean", RepDataForMLS[10]
},
{
"ZAcc_var", RepDataForMLS[11]
},
{
"ZAxx_sd", RepDataForMLS[12]
},
{
"bi_diff", RepDataForMLS[13]
},
{
"bi_sd", RepDataForMLS[14]
},
{
"class", ""
},
}
}
},
},
GlobalParameters = new Dictionary<string, string>()
{
}
};
Debug.WriteLine("Created Data");
const string apiKey = "YourKey"; // Replace this with the API key for the web service
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey);
client.BaseAddress = new Uri("https://ussouthcentral.services.azureml.net/workspaces/4d9fb0e8f83b47bb9ed3630de257eed4/services/b57e98ec4fdd4b43aad40030b05b943e/execute?api-version=2.0&format=swagger"); // WARNING: The 'await' statement below can result in a deadlock
// if you are calling this code from the UI thread of an ASP.Net application.
// One way to address this would be to call ConfigureAwait(false)
// so that the execution does not attempt to resume on the original context.
// For instance, replace code such as:
// result = await DoSomeTask()
// with the following:
// result = await DoSomeTask().ConfigureAwait(false)
var json = JsonConvert.SerializeObject(scoreRequest);
var content = new StringContent(json, Encoding.UTF8, "application/json");
HttpResponseMessage response = await client.PostAsync("", content).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
try
{
string result = await response.Content.ReadAsStringAsync();
var json_result = Newtonsoft.Json.Linq.JObject.Parse(result);
Debug.WriteLine("Result: {0}", result);
string next_entry = string.Format("Rep {0}: {1} \n", RepCounter, json_result["Results"]["output1"][0]["Scored Labels"].ToString());
lblSetData.Text = lblSetData.Text + next_entry;
RepCounter ++;
}
catch (Exception e)
{
Debug.WriteLine("error");
}
}
else
{
Debug.WriteLine(string.Format("The request failed with status code: {0}", response.StatusCode));
// Print the headers - they include the requert ID and the timestamp,
// which are useful for debugging the failure
Debug.WriteLine(response.Headers.ToString());
string responseContent = await response.Content.ReadAsStringAsync();
Debug.WriteLine(responseContent);
}
}
}
In conclusion, IR sensors and accelerometers were used to collect exercise data of the user wearing the shirt. The data was sent through bluetooth to a phone application created in Xamarin where it was stored. After the user was done an exercise the data would be processed to extract information about each rep within the exercise. Each rep data was then sent to a machine learning algorithm to be classified as either a good rep, bad form, or a half rep.
Comments