Author Archives: Michael Froomkin

I described recently how law.tm got hacked and what I did about it.

Since then I’ve been in correspondence with Netnames who argue pretty convincingly that the problem was not on their end. They say their records show no DNS changes for the period in question, and no one else they serve has has had a similar problem.

That means the DNS change happened at my hosting company (or, theoretically, somewhere else via cache poisoning, but that’s really unlikely). Hacking my DNS via my hosting company is certainly possible, even though they don’t manage the registration. That said, it’s sort off an odd thing to think happened because any hack capable of making that change there should have been able to do far far more damage and hit at least the other domains I manage from the same machine. Indeed, depending on the vector it could theoretically have hit all of them: my domains are spread over three machines; changing the DNS would require either root-like access on one of them or access to a control panel that gives some power over all of them. Yet none of those things happened. Maybe I dodged a bullet.

Meanwhile, I’ve changed all the relevant passwords (which were already strong random ones) and am working hard to plug every hole that my host’s automated security scan says it identified. Unfortunately, I have a lot of sites covering a very wide variety of personal and professional projects that have grown up over the years, and the scan resulted in a 12-page single-spaced list of things that might need fixing. It correctly identified some outdated installs of software packages, but the list of so-called hacked files seems overwhelmingly to consist of false positives (I’ve been investigating them with a simple text editor and so far they are mostly simple HTML files created by my cache program and fitted with legitimate headers) — so this is not an easy job.

Starting sometimes late Friday, my personal web page at law.tm stopped returning my boring homepage, and instead produced this:

There was also a sound track, produced by embedding this youtube video of “Epic Anonymous Rap song – Hackers” several screens below the main image:

Naturally, I wasn’t pleased, even if I sort of liked the middle part of the rap. Why attack me of all people? I’m for net freedom. Worse, the hack was blocking my main personal email address. Still worse, I was no longer able to access the domain via sFTP or ssh — everything timed out — making debugging somewhat challenging.

Eventually I figured out another way to log into the host machine, and verified that none of the files on the law.tm domain, including the .htaccess file, had been changed. This removed the most likely vector of the attack. That left two possibilities: The first was the very unlikely possibility of some very subtle SQL injection attack plus a level of traffic so hosing the domain that I couldn’t get through to it via ssh; this seemed unlikely because if there really was some DDOS-like event in progress I would have heard about it from my hosting company, Dreamhost, when the machine crashed, plus the redirect of the web page shouldn’t have worked either.

That left option #2 as the main suspect: a hack of the DNS records. The DNS records for this particular domain are manged by a different company than my web host, and their help desk is (a) located in London and (b) only open 9-5 London time Monday-Friday, leaving me high and dry for the weekend (smart hackers?). Perhaps coincidentally, perhaps not, I had just renewed the law.tm domain a few days earlier with the Netnames registrar. So the only thing I could do while I waited for the Netnames help desk to wake up was try to satisfy myself that this really was a DNS hack. That proved harder than I would have liked: the DNS records seemed to show incorrect information, with web requests for the domain being pointed to 80.249.100.3 and mail being sent to 75.102.13.215, neither of which was right. But then again sometimes the nslookup would come up ok with the right data. It could have been a propagation issue but why then were my http requests, even when I cleared DNS cache, never going through to my real page? Maybe, I worried, I didn’t know how to read the DNS records properly.

So I struggled with the problem. On Saturday I felt hamstrung by unusually slow and poor helpdesk support by Dreamhost, who have been much better in most of my past interactions. This time they announced a new-to-me policy that we couldn’t communicate by phone, only email, as they want a written record of anything relating to a security issue. And it took hours to get the first email response. When they did swing into action Dreamhost also refused to confirm I was having a DNS issue, even though that would have gotten them fully off the hook, saying only that the results were “ambiguous” … although in retrospect, that may have been an accurate assessment … so maybe score one for them after all. Unfortunately other than giving me an automated scan that showed possible problems elsewhere in things I manage but not on law.tm or its users. they didn’t say anything helpful about what else the problem might be.

In the end, thankfully, the problem seemed to solve itself this afternoon. The dig and nslookup data changed for the better — no more signs of the 80.249.100.3 or 75.102.13.215 IP numbers. OpenDNS’s cache started reporting the right info in more and more locations. Pretty soon all was back to normal. I even got a few — so far, sadly just a few — of the test messages I’d sent myself. (If you emailed me Friday evening or later, send it again please).

So I’m now pretty sure it was a DNS issue. Whether netnames got hacked (it’s happened before), or whether it’s some particularly ham-handed activity in connection with the domain name renewal, I may never know. Everyone I used to know at Netnames, which has been taken over once or twice since I last looked, seems long gone.