Software apps and online services
Hand tools and fabrication machines
FormCheck is a system that detects errors in people's form during strength training exercises. When the user improperly lifts, FormCheck will give audio feedback to notify the user on how to correct his/her form. FormCheck reduces training injuries, chronic diseases, and financial costs related to hiring personal trainers.
FormCheck is incredibly easy-to-use because there is no set-up or wearables that the user has to put on. Simply step onto the platform and start lifting.
There are no other products on the market that detects and corrects people's exercise form. (also, this submission is for the health & fitness category)Why Is FormCheck Needed?
What is the leading of death in the US? Answer: chronic diseases (heart disease, cancer, stroke, diabetes, obesity, etc). In fact, seven out of ten Americans die from chronic diseases. Luckily, we can significantly reduce our risk of most chronic diseases through exercise. Some studies even suggest that physical activity can reduce the risk of breast cancer by 75%, heart disease by 49%, and diabetes by 35%. If the incentive of reducing your risk of chronic diseases isn't enough to get you to exercise, how about saving $2, 500 per year in health care costs?
Most experts suggest a balanced exercise regimen consisting of both aerobic and muscle-strengthening activities is best for our health. For instance, the Centers for Disease Control recommends that adults perform muscle-strengthening activities at least two times a week.
However, many strength training exercises, done incorrectly, especially with added weights, can lead to acute and chronic injuries such as herniated discs, Tendinitis, stress fractures, and dislocations. Therefore, for those who are just starting to get into strength training or are already training but want to add on more weight, it's of utmost importance to have proper form.
One way to check if you're lifting correctly is to hire a personal trainer. However, for many of us, this is a serious financial burden as the average personal trainer costs $80 to $125 per hour.
Not being able to afford a personal trainer is a huge barrier to entry for many people who want to include strength training in their exercise regimen. It seems that we have three choices: either pay loads of money for an expensive personal trainer, go to the gym anyway and risk getting training-related injuries, or simply just not strength train at all and risk getting chronic diseases.
This dilemma is faced by millions of people in the US alone. The CDC estimates that at least 23.5% of American adults do muscle-strengthening activities regularly.
If there was an easy-to-use and affordable system for learning how to properly do exercises, then more people would start strength training and avoid getting training-related injuries while reducing their risk of chronic diseases.System Overview
The system consists of a Xbox One Kinect 3D depth camera to measure and calculate the positions of the squatters' joints, a balance platform with custom electronics underneath to provide weight distribution data for the program running on the computer, and a computer to process the data and give the squatter feedback on his/her form.
Detailed descriptions of each subsystem are included towards the bottom sections of this article.What is FormCheck Currently Capable Of?
Currently, FormCheck can detect and provide feedback for most form errors related to squats. I chose squats as the first error to detect for because squats are one of the most popular exercises, yet they are one of the main causes of lifting-related injuries. Even professional subelite powerlifters report that 42% of their injuries are from squats.
When the user wants to use FormCheck, he/she will step onto the platform (FormCheck will not report errors when the user isn't on the platform to prevent it from reporting false positives when the user isn't squatting). Once on the platform, FormCheck waits until the user is standing still and then calibrates to the user's height and weight automatically. When it's done calibrating (takes about one second), it'll say "Ready to Squat" and the user can start squatting any time after this cue.
FormCheck can detect when the user is starting to squat and will start looking for form errors during the squat.
There are several squat form errors that FormCheck detects:
- If the user's knees cave inwards during the squat. This error is reported, when viewing the lifter from the front, the horizontal distance between the center of the lifter's knees is less than 20 cm.
- If the user squats too deep. This error is reported when knee flexion (angle between upper and lower parts of the legs) for both of the knees at the bottom of the squat is less than 65 degrees. This "error" is actually pretty controversial in the lifting community and should only be enabled if the person has knee issues.
- If the user doesn't squat deep enough. This error is reported when knee flexion (angle between upper and lower parts of the legs) for both of the knees at the bottom of the squat is more than 100 degrees.
- If the user is putting too much weight on one of his/her legs relative to his/her other leg. This error is reported when, at any time during the squat, the user applies 20% or more force on the platform with one leg relative to the other leg. This is known as a "hip shift" and is hard to visually detect so that's why FormCheck uses the pressure-sensitive platform.
- If the user's back is rounding outwards. When viewing the lifter from the side, an imaginary line from the neck to the hip bone along the spine can be drawn. An error occurs if the lifter’s back rounds outwards such that at the maximum deviation from the back to the line is greater than 8 cm.
- If the user's heels are coming off the ground. This error is reported when, at any time during the squat, the user applies greater than 40% more force towards the front of his/her feet relative to the back of his/her feet. (Note: I forgot to film this error in the demo video)
I chose these thresholds with the help of a certified personal trainer by squatting poorly and noting down the threshold that a personal trainer would be visually able to tell that I was squatting poorly.
If the system detects a form error, it'll say an audio feedback cue on how the user should improve his/her squat form.
Currently, the possible form error feedback responses are:
- "Your knees are caving in. Try to keep your knees over your feet as you squat."
- "You are squatting a bit too deep. At the bottom of your squat, your knees should roughly make a right angle." (Again, this "error" is actually pretty controversial in the lifting community and should only be enabled if the person has knee issues).
- "You are not squatting deep enough. At the bottom of your squat, your knees should roughly make a right angle."
- "You are putting too much weight on your right leg. Try to make sure your weight is distributed evenly between both of your legs."
- "You are putting too much weight on your left leg. Try to make sure your weight is distributed evenly between both of your legs."
- "Your back is rounding. Try to keep your back flat while squatting."
- "Your heels are coming off the ground. Try to keep your feet fully on the ground while squatting."
If no form errors are detected, FormCheck will then respond with "good squat."
When the user is finished with all his/her squats, the user would get off the platform and the system will stop looking for form errors. If the user wants to use FormCheck again or if another person wants to use FormCheck, all they have to do is get back on the platform and FormCheck will recalibrate to that person's height and weight.Existing Products?
Surprisingly, I haven't been able to find any existing products that correct people's exercise form. If you find one, please let me know by commenting below.
The closest somewhat-related product is called Upright Go. It's a wearable device that is adhered to peoples' upper backs to notify them when they've been slouching for too long so that they can fix their posture. However, it's a wearable device and it's not used for exercises.
Another product that's somewhat-related is called Perch Fitness. It's made by a startup and it doesn't seem to be on the market yet. Similar to FormCheck, it uses a 3D depth camera and can be mounted onto gym equipment like squat racks. However, it's main purpose is to measure velocity of the bar for mainly squats and deadlifts. To my knowledge, it cannot give feedback about a user's form, it only works with exercises that includes bars, and it's targeted towards college and professional athletes rather than newbies.
Because there are no existing products on the market that improve peoples' strength training form, FormCheck is not an incremental innovation and can create a brand new market.Software Overview
The software running on the computer was written in C++ with the Kinect for Windows SDK 2.0 (I talk about the microcontroller's program in the "Electronics/Hardware Overview"). The Kinect is surprisingly powerful with its ability to do gesture, emotion, and voice recognition. However, I was only interested in its ability to calculate joint positions in 3D space.
An open source alternative to the Kinect for Windows SDK 2.0 is libfreenect2 which is part of the OpenKinect project.
You can also use any 3D depth camera, but the code would have to be customized for it.
The program that runs on the computer follows the logic in this flowchart. The word "board" refers to the balance platform. I reused some of the code from the BodyBasics example provided by Microsoft in the Kinect for Windows SDK 2.0 (mainly - the creation of an application window and displaying the joints).
I wanted to use a Text-To-Speech API like Google Cloud Platform's TTI because their WaveNet technology is one of the best right now. Also Google gives developers 1 million characters per month for free. I couldn't find much documentation on how to use the GCP TTS API with C++ on Windows 10. However, I thought of an easier way that would actually make the system more robust. If it used the API and the internet was slow, then there could be a significant lag between when the form error occurs and when the program tells the user how to fix it, making the user experience much worse. Also, because we're only checking for a couple form errors, the set of audio feedback responses is finite. Therefore, it would be much easier and robust if we just store the audio feedback locally instead of sending it to Google every time. So, I just used the command line by making a HTTP POST request and stored the audio as WAV files.
3D Depth Camera vs.Pose Estimation Neural Nets On Webcams: I briefly considered pose estimation libraries like OpenPose so that I could just use a web camera to calculate positions of the users' joints. A project on Hackster called YogAI used OpenPose to guide yoga poses (I did not see this project when I developed FormCheck though). There are three huge pitfalls for current real-time pose estimation neural nets: First, they're very inaccurate compared to 3D depth cameras considering that 3D depth cameras are accurate to within 1-2 cm in all axes. Second, to get 3D position estimations using libraries like OpenPose, there needs to be multiple camera angles that are synced up which would use a lot of space in a gym. Third, they're very computationally-intensive. The YogAI project authors say that they could only get 0.23 fps with a 15 second lag on a RPI3. They could have used a faster embedded computer, but that wouldn't increase the fps enough to catch very slight form errors in someone's squat. For example, if a squatter's knees cave inwards, they're only inwards for about 0.5-1 second which would be easy to miss if the fps is low. Also, the user would want feedback on their squat right after they finish it, not 15 seconds later. They were also only using one camera so they could only get X and Y coordinates, but Z coordinates are also important for exercises and would require inference on multiple camera views which would increase the amount of computation significantly.
(I would like to note that I think the authors did an AMAZING job with YogAI and I don't mean this to be a criticism of their work. I am just explaining why I did not use pose estimation neural nets).
The actual source is at the bottom of this post (fair warning: the code I wrote breaks a lot of coding style guidelines because I was just trying to get something working quickly).Electronics/Hardware Overview
On the bottom of the balance platform is actually a steel frame taken from a used Nintendo Wii Balance Board with custom electronics. Even though this balance board was marketed as a toy, the frame is very strong and the load cells are decent. At each corner of the frame there's a load cell. I just used the ones that came with the balance board.
This custom PCB takes the resistance from the load cells, converts it into a voltage, amplifies the voltage, converts it into a digital signal, and sends it as a space-delimited string via serial to the computer. It also only establishes a serial connection with the computer when it detects a person is on the platform so that the program on the computer knows when someone is on the board. The moment someone gets onto the platform, the ATMEGA328P microcontroller starts waiting until the person's weight stops shifting and then calculates constants to normalize the pressure data and then sends the normalized data to the computer. I use the normalized data because if I used the raw data and the user was not directly centered on the platform, FormCheck would report imbalance errors even when the user does a balanced squat. These normalization constants get overridden every time a user steps onto the board.
This custom PCB is essentially an Arduino Uno with load cell amplifier breakout boards attached. You can easily remake this with an Arduino and a breadboard using the schematic provided at the bottom of this post.
The code running on the ATMEGA328P is also at the bottom of this post.Mechanical Overview
Mechanically, this project is very simple. I just built a frame to hold the computer and Kinect at a height that allows the camera to get a full view of the users' bodies. This entire frame design will be scrapped in the second version/prototype of FormCheck because I realized that it would be better to create a smaller device that could be mounted anywhere (read the "Next Steps: Future FormCheck Goals" section).
Originally, I was going to make the frame out of aluminum extrusions like below, but I quickly realized that aluminum would be too expensive especially when wood would be a fine replacement.
I redesigned the frame to be made out of 2x4 wood. The total cost of the wooden frame was about $17 compared to about $100 for the aluminum frame. Also, the black layer above the wooden platform is supposed to represent a foam workout mat, but I didn't use one in the actual demo because I couldn't justify purchasing a brand new mat for this project just to toss it out afterwards since I knew that in version two I wouldn't use a mat.
I created the 3D model of the frame before the Kinect arrived so I didn't know how high or far to place the Kinect camera from the platform. I followed the guidelines created by Microsoft below for using the Kinect as part of the Xbox One system.
The guidelines recommend that the Kinect should be 0.6-1.8m (2-6ft) above the ground and 1.4-3m (4.6-9.8ft) away from the person. After testing with the frame, however, I realized that it would be better for the Kinect to be lower to the ground to get more accurate lower body position readings.How to Build It Yourself
These instructions are pretty brief, but that's because even though it took a while to create FormCheck, it's pretty simple to reproduce it especially because I provide the code, schematic, and CAD models.
Step One: Gather the materials listed under "Things"
Step Two: Download the code and schematic at the bottom of this post.
Step Three: Assemble the electronic components according to the schematic provided. I don't list all the values of the capacitors and resistors, but if you Google "How to build an Arduino", just use the values given there. Alternatively, you can just use an Arduino and connect the HX711 load cell amplifiers to pins 2, 3, 4, 5.
Step Four: Either disassemble a Wii Fit Balance board or create some sort of platform such that the four load cells are located at opposing corners of the board. See my picture under "Electronics/Hardware Overview."
Step Five: Connect the HX711 amplifiers to the load cells.
Step Six: Download and install Visual Studio 2017 and Kinect for Windows SDK 2.0.
Step Seven: Open the FormCheck.sln file in Visual Studio.
Step Eight: Connect the Xbox One Kinect and Arduino/USB Type-B connector to your computer.
Step Nine: Compile and upload the Arduino code to either the Arduino or the ATMEGA328P.
Step Ten: Run the program in Visual Studio.
Step Eleven: An application window should pop up. If your body is in the view of the Kinect, your body should be represented as a skeleton on the screen.
Step Twelve: Get onto the board. Make sure that the Kinect can see your entire body. Your computer speakers should say "Ready to Squat" when ready.
Step Thirteen: Start squatting!Next Steps: Future FormCheck Goals
Now that the concept has shown that it is possible to accurately and reliably detect squatting form errors, the next step is to expand FormCheck to work with more than just squats. This improvement should not be technologically difficultbecause it's simplyjustadding more test cases, though, it will take a lot of trial-and-error to fine-tune it to make it just as good as a personal trainer. The next exercises I would like to implement corrections for are: deadlift, bicep curl, push up, and bench press.
Before implementing corrections for those additional exercises, however, I would like to consolidate the hardware into a single, maneuverable, package. I imagine that a commercial version of FormCheck would include the 3D depth camera, embedded computer, and speakers all in a single device. This device would also be able to run on battery or plugged-in. I envision that this device would be able to wirelesslycommunicate with accessories, such as the platform or wearables.
If I make it past stage one of this competition, I will create a "version two" FormCheck prototype. This prototype would be a lot closer to an end product as it would probably use a smaller 3D depth camera such as an Intel RealSense or an Orbbec Astra. For the embedded computer, I would need one with USB 3.0 and a decent CPU such as an ODroid XU4, Nvidia Jetson Nano, Google Coral, or Aaeon Up Board. These devices can all be powered by 5V USB so I would just purchase a power bank since they're cheap and have the charging circuitry already built-in. Lastly, I would 3D print an enclosure that would have swappable mounts so that it could mount onto tripods, sit on flat surfaces like tables, or mount onto gym equipment like squatting racks.
The balance platform could also be powered by a separate power bank and use a Bluetooth module to communicate to the embedded computer.
By making these improvements, the wooden frame, laptop, and wire between the platform and computer would no longer be needed. Further, since it would be able to be mounted almost anywhere, it could be used for many more exercises besides just squats.Commercial Viability & Potential Business Model
In the "Why is FormCheck Needed" section, I have shown that there is huge worldwide need for a product like this.
In the "Existing Products?" section, I have shown that there are no existing products like this on the market.
In the "Next Steps: Future FormCheck Goals" section, I have described the physical and software improvements that I would like to make in the near-future to bring it one step closer towards commercialization.
Now, I would like to share my vision of how I think this product would fit into the market and provide an accessible solution for many people quickly. FormCheck's initial target market would be gyms - not individual consumers (this is a B2B play not a B2C). Selling to gyms would allow me to initially bring the prototypes to a gym to have all their gym members test it out and give me valuable feedback. Theoretically, I could create a single prototype and have thousands of people test it at a single gym. Also, because this is a product that would allow personal trainers to work with many clients at once, selling FormCheck to gyms would give many personal trainers access to it.
Further, as anyone who has ever tried to bring a complex hardware product to market will tell you, "hardware is hard." One of the main reasons for this is the experience curve, which is a graph that shows how manufacturing a few products makes the cost per product very expensive, but manufacturing many products drops the cost per product quickly. The components of this product are already rather expensive with an embedded computer costing at least $50 and a 3D depth camera costing at least $150. The total cost of creating the product, assuming low volume, would probably be in the $300 range. Selling a product to individual consumers that's several hundred dollars would be pretty difficult and some may choose to just hire personal trainers. Conversely, selling FormCheck to gyms would be easier because they would have the money for it and it would complement their business model, enabling them to give personal training sessions to more gym members and drive more people to their gyms.
Theoretically, I would provide the hardware to gyms for free, making it a zero-risk investment for the gyms. This would be a Hardware-as-a-Service business model where the gyms would be charged a monthly subscription fee OR be charged a rate based on the number of users at their gym who use it.
What would be really cool, but kind of far-fetched, would be to integrate FormCheck into health insurance programs. Users who regularly use FormCheck should get lower health insurance rates because they're at a lower risk of chronic diseases since they are exercising and because they're at a lower risk of getting training-related injuries while exercising.