ScratchDB Hiatus
Unfortunately in May 2024 ScratchDB's server experienced an unexpected power loss at just the right time to cause database corruption. Since this server was meant as a temporary transition, backups were not routinely made. To add to the issue, the server currently is powered down in an area where it is difficult to restore power the foreseeable future. Unless some contacts are made, the earliest power will be restored to this system is mid-August 2024.
With this information, I unfortunately have to announce that the ScratchDB project will be on an indefinite hiatus, meaning service for v2 or v3 will not be restored, possibly forever unless certain circumstances change.
ScratchDB has been running since September of 2019, and since then has gone through a lot of changes. It started simply being a forum historical archive, then adding forum search, and then a database with projects and users. Its biggest change was the addition of rankings and then rankings over time. When I first started this project fairly early in high school, I had plenty of time to develop it and improve it with user feedback, but unfortunately, now being in my second year of college, it is rather difficult to maintain.
Some Notes
I often read feedback asking why I couldn't just run it in Vercel or Replit, and some people tried to replicate it there. The unfortunate truth is ScratchDB required a lot of processing power and orchestration, to the point of where it took up some 64GB of RAM to run with database caches and different temporary states. In total, the databases weren't too big though. Excluding project JSON information, the system was somewhere under 100GB, although this information had to constantly be updated which meant the system constantly had to re-check for updates. Storage was never an issue; for the past few years my servers have been running multiple drives over 8TB each. However, the difficulty comes down to stability and uptime.
The server ScratchDB was running on was a Dell R420 with 4x8TB and 128GB of RAM. This system was quite overkill for the project, but I've always had an interest in maintaining server equipment like this. I also didn't use this server exclusively for ScratchDB; I run a lot of home automation systems as well as other personal projects on it. I planned on moving ScratchDB to the new server that this current landing page is running on, which is an HP DL360 G9 in the FMT2 datacenter in Fremont, California. However, due to power difficulties, I wasn't able to make the transition in time.
ScratchDB was always a learning experience and a hobby, much like me trying to maintain these servers. Unfortunately, with learning comes making mistakes, some of those being quite difficult to fix. Database corruption due to power loss happened because I didn't follow certain good practices when I was setting up the temporary system. I didn't really think that would happen to me, but it did. Perhaps some bad luck was involved, but as some have speculated, the project couldn't keep running forever. ScratchDB has been an amazing learning experience for me as well as people who used the project.
When I first got the idea for ScratchDB, I simply wanted to see what forum posts got deleted to try to identify some biases in how moderation was done, but the project very quickly evolved into a project that much of the community used with a much more expanded set of uses. I never expected it to get this big, especially since it was just a database and an API with no front end to directly interface with it, but projects such as ScratchStats and Ocular came in and made good use of the data. It even grew to the point of where I heard that some ST members were using its search functionality. Unfortunately on the flip side I was also banned from Scratch... not too sure what that's about.
I'm thankful for being able to run the project for so long, even though there were some rough patches in the middle there and it got pretty messy in the end. This project has been on my resume for quite a long time and possibly helped me get into college and possibly helped me get my internships the past two years.
If you would like to get in contact with me, I can be reached at dat (at) lefty (dot) one, or on other external chat platforms. I may not respond since I have quite a lot going on currently, but I will hopefully eventually get to responding at some point.
The Future
ScratchDB is one of the few sources that has historical forum and user data, and I never wanted that information to go into the dark. I will keep trying to restore the database so I can get a dump of them. Once I am able to recover a decent amount of information, I will provide download links to certain chunks of information, such as follower counts and all non-deleted forum posts and topics.
In order to not violate certain privacy laws and practices, I will not be providing the ability to obtain deleted forum posts or forum posts' historical edits since some of those may contain personal information or information people want deleted. I believe in the right to have your data removed from the internet, however I am unable to accomidate deletion requests after the archive is made.
In order to ensure relevancy and accuracy, I will likely re-sweep the forums, so in order to be excluded from any database dump (even though something like the Wayback Machine already has it) your posts will need to be modified on your side.
Conclusion
As I'm writing this I am sitting in my apartment in California for my current summer internship. I never believed I would make it to where I am today, and ScratchDB definitely helped me get there. From getting me into Georgia Tech, to getting me a freshman internship at a FAANG company, I'm not too sure if the project directly helped or just gave me the skills I needed to get here. But thank you for using ScratchDB, and hopefully you've learned something from it as well.
I don't want this to be the last project I do involving large sums of data, so whenever I start something new I'll update this page with what's going on. I may also post a more detailed write-up of how ScratchDB works. But for now I need to focus on my internship and my school work.
Scratch on!
-Lefty
Updated 19 June 2024, 17:15 PSTRelevant Updated Information
ScratchDB (Forum Archive Only) Status Report================================================
Known forum posts: 8219281
HTTP response 200: 6999482
HTTP response 404: 571763
HTTP response 403: 648036
Current seeked ID: 8219281
Max post ID seen: 8219281
Saved forum pages: 847824
Searchable posts: 7001142
================================================
Report generated 2024-11-07 00:03:49.916868473 UTC
(200 = green, 404 = blue, 403 = red, unknown = black)