Skip to main content

Downtime and lost data

Posted by Kim on December 5, 2018, 9:55pm

Tonight was a hard night. The site was down for several hours for maintenance after our database went kerfluey. The issues we experienced were non-malicious (This was not an attack. No data was stolen.)

Most things came through the crisis just fine; so far, it looks like I was able to recover all groups, forum posts, private messages, kudos, accolades, custom templates, and purchase records.

Character and user pages DID take a hit, though. We have had to roll back character pages and users as much as ten days. If you created a character or account, or made changes in that time span, it's likely gone now. I can't tell you how upset I am that this has happened to our community, and how sorry I am.

I can say I'll move heaven and Earth to make sure this storm can NEVER happen again. I can also say I'll do everything I can to reunite people with lost data wherever it's possible - oddly, with your help, it may be possible re-link you to any RPs you were engaged in if you were doing them with an account or character fresher than the rollback. Read on for more details about that.

A LOT of data has been moved around to get us back up. There may be some unexpected results. PLEASE submit bug reports about anything that looks off to you to, no matter how small - you are never bothering me by bringing my attention to something that might be wrong, even if it turns out not to be a bug or problem! And you may well be helping me to help not just you, but many others, if you turn out to have spotted a clue to an issue. I really need your help to make sure everything is okay going forward!

Since user accounts were rolled back 10 days, I'm going through the purchase records and re-applying any Epicness or other perks you may have applied to your account in that time. This will take a few hours to sift through, and even so, you're still welcome to submit a bug report to me about it right away if you're worried!

If you lost a character or your user account in this awfulness, I could not be more sorry and upset on your behalf. The RPR is supposed to be here to support and celebrate your creations, not to hurt them. I want to do everything I can to at least re-unite you with the RPs you may have had.

If you have an email notification about a RP you were doing, whether in PMs or on the forums, please forward it to me at admin@rprepository.com with the name of your new account and and/or your re-created character and I will re-link that RP to your new account, and throw in some free stuff to try and make it up to you.


Don't have any emails about your RPs from us, but still know you had a RP that you want to be reunited with? Still email me with as many details as you can and I'll see what I can do. Include:
  • Your username
  • If applicable, your re-created character's page (link me!)
  • Any details you can remember about the RP you were doing. What forum was it on? What was the title of the PM conversation? What date did it start? That sort of thing!

I understand if you are feeling anger or sadness or even grief in this moment. I am feeling these things as well. It's an awful thing to happen and I am certain we all feel awful about it.

I want to thank everyone who sent me SO much support while I was working on this issue. You got me through it!

I will continue to update this space with any critical updates, as well as put out notices on our Facebook and Twitter feeds.

UPDATE 12:30am PST: I've restored hundreds of thousands of missing character pages. If you're still missing something more than 10 days old, definitely make sure you hit me up with those bug reports!

UPDATE 12:52am PST: If your account just adopted a character named "Zombie character," this is a placeholder for a lost profile. It is intended to give you back access into your private messages and groups you were participating in with that character. You may choose to delete the character, or to rename it and rebuild the profile on it.

UPDATE 1:30am PST: If you activated an item in the last ten days, the benefits should now be re-applied to your account. That is to say, your Epicness and character slots should be back.

Comments

hibiscustea

December 6, 2018
5:16am

Love you guys! <3

TimTowtiddy

December 6, 2018
4:37am

Sorry to hear that you guys had a tough night, but thank you for your hard work and dedication to getting us all back up and running.

I've worked in IT for many years; if you want someone to help maintain things (especially the db) please PM me. Depending on how you have things set up, I might have ideas that could help.

Thanks again.

TheCreator7

December 6, 2018
4:36am

A big squishy hug goes out to all who worked on the site, those who lost things and in particular Kim.

I learned the hard way when another RP site crashed a great many years ago (and it was around this time of the year as well) to always back up stuff on a portable hard drive as well as on my laptop. Or at least on a flash drive. My laptop went down at the same time so I had a horrid time recovering my stuff.

If you were unfortunate to lose characters and information, try Google. Type in your character name and see what you can find. IE: Rosewoode Cara Deforrest (My first Character ever made)

kUylHYa.jpg

As you can see, Google does keep track of your characters but you have to think of where your character was and perhaps you will be as fortunate as I was to recover things that I had forgotten.

I wish all of you who did lose information a speedy recovery of your information!

Kidd

December 6, 2018
4:12am

It's a shame what happened, and a hard lesson for everyone: back up your characters! RPR allows you download PDF copies of your character pages (usually I only do this when I voluntarily delete a character but now I'll do it with all my favorites).

And thank you so much for your hard work in restoring characters and their pages!! We all appreciate how hard you work to provide RPR as a service to us--often with no cost to us. I'm super grateful for RPR.

Rogue-Scribe

December 6, 2018
4:06am

Big Hugs for you Kim. Thank you for working so hard to get things righted! My initial assessment is I lost a recently created character and some significant edits to another character, with only minor edits lost on a couple others. I'll send you the details on the character in a bug report.

I have to say that your love and dedication and hard work put into this site is greatly appreciated by all of us who come here to play. Thanks again for everything! :)

Hades_

December 6, 2018
3:58am

Everyone dropping comments here about missing characters, PLEASE send it in as a bug report for kim to keep track of everything.

She has done so much work tonight and done so much recovery for one person and wow.

Bug reports folks! They will be the best help!

Pineapple

December 6, 2018
3:55am

Kim thank you, you are amazing. <3 I'm sure everyone on-site appreciates all the hard work, love and I'm sure in times like these tears you put into RPR. Hope you got some well deserved rest!

Taciturn

December 6, 2018
3:55am

That was quite a few scary hours!

I'm glad it's only some data that was lost and not all of it, I'm sure it could have been a lot worse if you weren't working so hard on it.

lagenas

December 6, 2018
3:51am

Darn... this is a bummer I lost one of my newest characters involved in several RPs... I understand stuff like this happens sometimes and that it's very stressful on all of the staff. We all really appreciate how much effort you guys put into the site!

Luckily I had one of the pages for my character pulled up, so I was able to get about 50% of her info back but I unfortunately didn't back up the rest elsewhere. :( If you guys are able to recover anything for a character named "Dahlia" under my account that would be really helpful, but if not it's fine!
I'm glad you guys were able to restore chats with the nuked characters in them at least haha.

Abigail_Austin

December 6, 2018
3:49am

*hugs for Kim* Everything seems to be okay on mine. A relief!

tear_ps

December 6, 2018
3:34am

I lost my latest character, Rayne Crimson, and I don't have her info! :firefly-is-over:

Vali99

December 6, 2018
2:50am

One of my latest characters is gone, is this part of this? Like fully gone and replaced with 'Zombie character' I put a lot of effort into that character and I would like to know what happened to the character 'Captain Jacob Skye' and if any other user has this problem.

Pirate

December 6, 2018
2:24am

We appreciate what you're doing. You must've recovered the site in record time! Please, take all the time you need. We understand that such issues can't be solved in a couple of hours.

Kim

December 6, 2018
2:21am

Thank you again for all your words of encouragement through this ordeal!

It's now 2:21am and I HAVE to sleep. I'll pick up again in the morning to help in any way I can. <3 Love you all!

Mars

December 6, 2018
1:25am

That was probably the quickest recovery I've seen made in so little time! I sense that enormous and hardworking efforts have been made (and probably continue to be made)! Thank you for working so hard!

DaBoisBackHome

December 6, 2018
1:16am

Thanks for working so hard and long to get things working again. Really appreciate it, even if I lost a few things. Thank you so much. Now go get some rest, Kim. You deserve it!

RedHeadSamurai23

December 6, 2018
1:11am

Take your time. We all appriciate your effort 100%.

GamerWoona

December 6, 2018
12:30am

That gave me quite the scare when I tried posting an answer yesterday night. Good to hear that things are back in control again! It is also nice to see how positive everyone seems to be about this so take a little pride in what you built here, Kim :D

Dndmama

December 6, 2018
12:27am

You're doing an amazing job, Kim. Miraculously it looks like I didn't loose anything but a small status update on my profile which I've since fixed myself. We all appreciate the hard work you're putting in. Make sure you take some time to rest and recover once this is all done with. You deserve it.

GalaxyStar2

December 6, 2018
12:24am

Thank you,for rebooting this site Kim,your the best :)