Applying machine learning to chess scoresheets

Hello, who are you and what are you working on?

Hello! My name is Rithwik and I'm currently a freshman at UC Berkeley. My interests have varied more than I thought was possible, from writing to astrophysics and now to programming. That said, my whole life I've wanted to be an entrepreneur—ever since I started worshipping Steve Jobs as a 5th grader while trying to convince my parents to buy me an iPad.

Alex Fung is my cofounder, now also a freshman at Berkeley. I met him as a freshman in high school and we grew to become close friends over the years. At the time, I was the President of our school's chess team and I desperately wanted more members, so Alex decided to join.

Despite losing the championship finals match that year, I found a bright side to the mess: we came up with the idea for Reine. What if I could scan all of these chess scoresheets and have them analyzed by my phone? FYI, chess scoresheets are handwritten documents used to write down the moves of a game, often later entered into a computer to learn from your mistakes. It's especially brutal to have to do this just after you've just lost the most important match of the season, and I was just dabbling in machine learning at the time, so everything clicked. Not only did it sound really cool to make a product that could scan handwriting (and implement my machine learning knowledge for the first time), this was also a problem that my team, I, and the whole chess league faced. Because I was involved in the chess community, everything followed through from there.

What were the early days like?

I spent many days in class writing out ideas for how each part of the product could work. The "Hello World" of machine learning is the classification of handwritten characters, so that part of the problem was solved for us by much smarter people. The difficulty was in adapting this into our product and making sure that it would actually be accurate enough. We considered problems like this over and over again, filling up more pages in my biology notebook than the biology notes did. At some point, we split up tasks and started coding out the ideas. It was super useful to have everything planned out because the only real issues we ran into later were with code debugging, not with coming up with logic from the ground-up.

How have you grown Reine's usage?

This is where it really helped that I had been involved with chess for the years before Reine. I was part of two chess clubs, one for the school league and one for the city. I was able to have people try out the scoresheets whenever I played matches with these clubs in order to test Reine. Seeing people getting excited about the idea alone was gratifying.

I knew Chess.com was the biggest hub on the planet for chess players, with 24 million users, so I reached out to Chess.com's CEO, Erik. He loved the idea and (after an initial no, after which I asked again :)) offered to post an article Alex and I wrote as the largest banner on their front page.

Beyond that, I had been interested in the tech scene (and still am) since my obsession with Apple. I knew Hacker News was where people first posted their newest project or idea, so I decided to make a post to our website on there. My friend also wrote articles for an AI blog on medium that paid money for written work, so I shot them an email too. Reine reached the top 3 on the Hacker News front page, and Heartbeat AI published a 4,000 word article on all the tech behind Reine.

Today, 10,000 people have tried out Reine (mostly from the HN post), 100 have commented on our various social media posts, asking myriad questions about how it works, and multiple people have offered to contribute.

How have you managed the workload?

Creating Reine during the first semester of senior year in high school was insanely stressful for us. Like the tens of millions of other seniors applying to college that year, we had essay after essay set out for us, and these essays mattered.

Luckily, it turned out that Alex applied to his favorite schools early, and by the time I started applying he had already finished. This made it easy for at least one of us to be working on Reine at all times. After applying (and being waitlisted to several colleges), we had all the time in the world and tons of motivation, so managing the workload at that point wasn't too bad.

How much time do you devote to Reine?

At this time, as college freshmen adjusting to a very new lifestyle, we don't have much time to spend on Reine. Currently, Reine can only be used online through our website by uploading an image of a scoresheet. Our next task to make this product usable is to have Reine run on smartphones. Unfortunately, neither of us currently know how to make apps, so we've put that project on hold.

What are the biggest obstacles you've had to overcome?

Chess.com made our post live on the front page within minutes of me letting them know I had written the article. I had assumed it would be at least a week before I would even get another response. Given that Reine wasn't fully working at that moment (although it was complete, there were bugs), I started panicking. I called Alex and asked him to work on fixing the server-side as he knew it much better than I did. I asked my dad to talk to one of his friends who knew how to run AWS websites so I could work on that as a backup for our free PythonAnywhere server. I was also trying to communicate with Chess.com, hoping they could give us some more time. Eventually, Alex got the server working, and I thought everything was fine. No more than five minutes later, our post on the front page of Chess.com was up, but the link was dead. It turned out that since I made a new Chess.com account without any games played, the system automatically assumed we were spamming the website. After several emails, fixes to the problem, and reoccurrences of the same problem, Chess.com eventually had to put up another post on their front page around 6 hours after ours was first up there. Despite this being a terrifying situation, our website link was thankfully visible as a subtitle from Chess.com's front page, so thousands of people were able to try out the demo. The week after I posted Reine on HackerNews and getting to the front page there as well boosted our morale right back up.

What are your hopes for the future of Reine?

Personally, I think this software can be used in many areas beyond chess. The core of why our product is highly accurate is because we have underlying information about the nature of the written material itself. For example, we don't just know the shape of the handwritten letters, we also know that if the first character is a letter, the second must be a number. Information like this can be used to make the next generation of handwritten forms accurately scannable. I truly hope that Reine becomes the next Scantron.

What advice do you have for other open source projects and maintainers?

Neither Alex nor I are very experienced with open source projects, this was only our first one, and yet, I think I can give one piece of very valuable advice for them. Originally, Reine was not open source, but after telling the internet about the idea, a few people asked if they could see the code and contribute to it. That's when we decided to release it, and I'd encourage others to do the same. If your idea truly is interesting, people will pour in at least asking questions about it. Seeing people ask questions and post comments on our various articles was ridiculously motivating. On top of that, you might get a few people that actually join the project!


To find out more about Reine Chess, you can visit their website or star the project on GitHub.

Editor's note: as a newbie chess player myself, I'm in no doubt that Reine Chess is solving a clear problem. Transferring your moves from scoresheet to computer is a huge pain, and I say this as someone who typically only plays one match every week or two. If you're a chess player yourself, or just want to work on an interesting machine learning problem and save chess players a lot of time, consider contributing to Reine Chess to make it even better.

Reine Chess