Washing Machine Checker App

The Washing Machine Checker App is a project that I and a couple of my friends did over the course of a 24 hour sprint for a hardware hackathon at UCSD called IEEE Hard Hack that took place in Spring 2023. I had brought some parts to the hackathon including my Raspberry Pi, my friend had brought a camera sensor that he owned and we had access to ultrasonic sensors as part of the hackathon materials. We wanted to come up with a project that would act as a solution to a genuine problem that we faced at UCSD, and it needed to fit into the theme of the hackaton, which was "sensing the world".

We did some brainstorming, thinking about what most inconvenienced us as students, and ultimately decided to do a project to address the annoying wait times and unpredictable availability for campus washing machines. We decided to acheive this by building a hardware system to sense when washing machines are in use and when they are free and then update this information on a web app (a system like this unbelievably does not already exist). For our underlying hardware, we planned to use the ultrasonic sensors in combination with the camera to detect when the machines were started, in use, finished washing, and unloaded of clothes (important that this was a different state than the machine merely being finished).

The early stage of our project mainly involved planning out how we wanted to structure the different components, setting up and updating the Pi so we could ssh into it, and getting the sensors from the materials room. The ultrasonic sensor code was implemented first since that was the easiest part of the project and then we started on the computer vision aspect, which was extremely tricky. None of us had used OpenCV before, or even tested the camera out, so we essentially dove into this part blindfolded.

We experimented a lot, pouring in several hours of our project time in trying out different ways to read the digits from the LCD screen displays of campus washing machines and dryers. We took multiple sample pictures of the displays, trying to use image processing functions of OpenCV in combination with a library called pytesseract. It was a massive struggle, but eventually we were able to get pytesseract to recognize text from a couple of our images that were centered correctly. The problem is that the digits on the LCD display were segmented rather than smooth, which made it hard for pytesseract's OCR functionality to determine that they were actually numbers and not something else. We had to perform several transformations on the images to get them into a form that we could use, including conversion to grayscale, blurring, finding contours, and warping the image to get a final result seen in the image below: a clean and thickened set of black digits on a white background, which we could then feed into pytesseract. Most of our computer vision code was written through finding examples on the internet, manipulating values and testing how it affected our image. We were not able to develop a perfect program in time and our file was still in very much a test phase by the deadline (seen in how the file is named blah.py), mainly due to the time-pressure of the competition and our limited experience. However, we certainly learned some interesting concepts about OpenCV and that computer vision is actually pretty challenging and should not be underestimated. For that reason, I am taking a computer vision class Fall 2023 to properly learn computer vision concepts that I can apply to other projects in the future.

A snippet of our code.

A view of our web app.

Our code successfully performing text recognition.

In addition to the computer vision code and ultrasonic sensor code, I also worked on the code for the backend portion of the application, which used threading to read the distance and duration of obstruction from the ultrasonic sensor in addition to the lcd screen reading, both of which would inform if the machine was started (display would change from price to time remaining), in use (display shows time remaining), finished or unloaded. For the finished versus unloaded state, the ultrasonic sensors came in most use because sometimes people would confuse a washing machine with someone else's clothes as theirs and open the door, only to realize their mistake and not unload the clothes. This would reset the washing machine display back to the price but the washing machine would still not be "available". To deal with this condition, the ultrasonic sensor would detect the duration that they were in front of the door for after opening the door to determine if they actually unloaded the clothes or not.

Overall, I had fun working on this project with my friends and enjoyed the thrill of my first 24 hour coding sprint, pulling a full all-nighter to get as much done as I could. These types of rapid-paced projects are challenging, but that's part of the experience. With all the experience that I have gained since then, I can't wait for Hard Hack 2024!