I did this once. I work for a music streaming service. Most websites and apps have one copy of the database that customers use for real, plus one or more other copies that developers can use to test stuff. The production database is usually hosted on a server owned by Amazon or another large service. Many developers keep a smaller copy with less data in it on our local computer.
In my case, I had written a small script that was meant to delete information from the small database copy on my computer, so I could refill it with more up-to-date information from the production database. I accidentally ran the script on the production database, and in seconds, I had deleted millions of songs from our service (not the actual song files, but the database rows containing their names and all the other information about them). That meant our customers could no longer listen to music.
Fixing this was not easy, either, because even though we have backups saved, restoring millions of rows of data takes a long time. It took over a week to get everything put back the way it was. It was a disaster.
Latest Answers