This week I had the most trying experience of my online career. I’d like to take some personal responsibility for it, and I will in a moment, but let me first say that MediaTemple is the worst host you could have the misfortune of dealing with and you should avoid them at all costs lest ye desire to suffer the same as me.
So, first, the story.
On Monday morning I got a notice from RSA on behalf of one of their european bank clients that I had some malicious script on my dedicated virtual server with Mediatemple. Shortly after that I got an email from Mediatemple saying the same.
I quickly opened up the plesk file manager and started looking for the offending files (the support request indicated particular directories). I found the files, deleted them, and then started checking file permissions for obvious problems.
Part of the way though the check my website suddenly said ‘Error establishing a connection to a database’… Ok, bad sign. I was also in the command line via ssh and I was kicked out of there as well.
I started a support ticket with Mediatemple and started an online chat to find out what was going on. I was concerned that command line change I made to the permissions of a folder might have caused this.
I’d like to point out that at this time Sandy was beginning to bear down on the East Coast.
I was informed by Rodel on Mediatemple’s instant chat that they were shutting down my server and I would not be allowed to access it again.
I asked why he couldn’t give me an answer really but informed, sadly, yes I would just have to fuck off. Bye.
Now here’s the part where I have some responsibility. I guess I should have had redundant backups outside of the dedicated virtual server I had with them. I guess Instead of using the snapshot backup that they provide for the whole server that I had just refreshed, I guess I should have had local files. In fact there are a whole slew of things that I could have had in place as contingencies but how could I ever know that they would simply shut my server off and tell me to beat it.
Back to Rodel and now enter Mike the ‘Abuse Engineer’ and their sorry jobs at this shitty company.
I called immediately waited a bit and then was able to speak with someone who informed me that I had violated their terms of service. I had uploaded malicious scripts to my server and I should go away. I spent the next 20 minutes pleading for the opportunity to simply get my files off their server. At first they told me ‘no’, I should have had backups that weren’t on this server. After another escalation and a lot of waiting and periodic power outages I was informed that they would allow me 24 hours access to the files that they would basically sequester in a separate directory.
This meant that I would not have the ability to do, for example a SQL dump or to access the databases in any normal way. I’m not going to get into resolution in this post, this post is about the abhorrent customer service of Mediatemple, the next post is about what to do when everything implodes and your hosting company is full of assholes.
As it turns out Mediatemple had actually sent me an email telling me that they believed I was guilty of abuse earlier that day so I could have had an opportunity to address this BUT the email was deep in my gmail SPAM and I never saw it.
So after 3 years with their overpriced inferior dedicated virtual server service, having never been late in payment, without any history of abuse in any way, despite the fact that I’m paying $135/month or whatever the hell it is. Despite the fact that the server was in my own name that they had my contact information and that I’m public and you can find me and I’m not some lurky Internet bad guy, they decide the best approach to the this relationship is to simply shut off my service.
Never mind the hundreds of paying customers I have. No don’t look at the site or the activity. Just shut me the fuck down.
Now that I’m at the tail end of the week I’ve had an opportunity to figure out how those scripts got onto my server.
I did it.
There are two possibilites. One is that I have something on my Macbook Air that is intercepting my uploads… meh? I doubt it. I deleted Filezilla just in case. The second is that I uploaded the script as part of the development process.
I’ve been experimenting with Jquery and helpers and libraries and I had uploaded a directory to a separate domain and IP form my primary site in order to test this particular plugin that had to be on a server because you can’t test cookies on your local machine.
So for the highly egregious act of uploading a jQuery plugin that I downloaded from a reputable site the bastards at Mediatemple thought the proper response would be to blow up my entire online existence without any real communication about it.
After 3 years did I get the benefit of the doubt? No.
Did they bother to look me up and see if I was legit and someone they could work with? No.
Did they offer to help me identify or lockdown whatever may have caused the problem? No.
No, instead what Mediatemple decided to do was to make what was a minor security threat into a massive business catastrophe for me and major inconvenience for hundreds of my customers and who knows how many visitors.
Not only is their service inferior to RackSpace and Amazon AWS and the like their customer relationship policies are horrendous.
In my next post (or two). I’m going to tell you about how we came back from the dead.
So where are the learning moments in this rant?
From a technical and procedural perspective I have a lot that I’ll share in the future.
From a customer service standpoint having now been victim to the worst customer service or support ever, I’ll say this. I know that in the future when I get alerts that customers are violating license terms or if something looks fishy I’m going to seek to communicate with the customer like a real person before I make any judgements that are going to impact their lives or service.
Second, I started feeling like MediaTemple sucked almost a year ago. I should have moved then. I should have listened to my intuition. Yes moving servers is a pain in the ass but its much less a pain in the ass when both servers are running, there’s no hurricane or power outage and there’s not some inadequately endowed ‘Abuse Engineer’ flexing his tiny little muscle against you.
So there are the lessons.
1) Treat your customers like real people, they fucking are.
2) If you’ve got a bad feeling about a company, you’re probably right.