Last week I was invited by an aquaintance to look at a web site which had been DDOS-ed. The attack itself hadn't been too problematic. So, that was that and this has become one of the most boring blogs in the history of mankind.

Well, it still might end up that way, but at least let me try to change that;

do upon others...

A couple of years ago I received an email at my abuse@ mail address where some unknown system administrator notified me that he had seen one of my IP addresses connecting to one of his and he wondered if I knew anything about it.
It turned out that a hacker had replaced the bash shell with one with a couple of more features. Without that guy helping me I would have never known.

So, I figured it was my time to execute a random act of kindness and go through 10GB of logging and see if I could notify other system administrators.
Some scripts were made, IP addresses were being looked up, domain names were stripped and uniq-ed, abuse@ was added in front, a nice friendly message stating the issue had been written and over 1500 emails had been sent.

conclusion near the end in case of tldr;

As my story turned out to be quite lengthy you can scroll down to read the conclusion. If your attention span exceeds that of a bored teenager, I do hope you enjoy reading it.

Anyway, almost directly my mailbox was being filled with mails being undeliverable as quite a lot of companies do not seem to have a working abuse@ email address.
I did expect that, but still was somewhat surprised by the sheer number of them.

After a while though the first real responses began to arrive. I have to tell here that I hadn't been able to add pieces of logging to my email, and that is what most people wanted. Soon my mailbox was filled with hundreds of emails from people requesting for logging and I started to feel very sorry for myself.
But still, I had a debt to pay and so I did go through the logging again, again and again again.

irrelevant random picture I happen to like

to lighten everything up.

large companies are weird

Then some stranger responses arrived, mostly from larger companies;

• One of them requested me to fill in a web form, which I thought was funny and jokingly I filled in the darn thing. I wrote that they could contact me if they wanted to through email and that I would help them or that if they wanted me to be part of their administrative process that I'd be happy to do that for a reasonable consultancy fee. I didn't expect any response, and I didn't get any. It wasn't my problem, I was buy enough helping others in my own time, so who cares.
• Another one returned my email with the notice that my ticket could only be validated after me calling a 0800 number. Living in another country I don't know if that'd cost me, but I never intended to find out, so I had to let that go as well.
• I really like the guy stating that I should use PGP with my communication. Although I can appreciate that, I wasn't going to bother so I replied that he could have it in clear text, or not at all, for free, or with PGP for a fee. He decided that having a secure channel was suddenly less important. I still see why he would prefer PGP, but hey, if security is an issue, be prepared to pay for it. On the other hand, if security is an issue, don't let someone compromise your system (I'm sure that'll come back to haunt me, oh well, I never claimed to be perfect, or if I did, you can't prove it!).

Then some patterns started to occur, probably a couple use the same kind of software and were very definitive in their reactions;

very weird indeed

• I should resend my email in the proper format or else they would not help me helping them. That'd teach me to bother them! So I replied them that they could have the information for in exchange for a 'pretty please' in the format of my choosing or for a fee in a format of their choice. Of course these emails were replied with the answer that I should resend my email in the proper format. Never heard from them again.
• Some other companies complained that they couldn't give me the information I requested because of privacy issues. I didn't request any information, nor was I planning to solve their problems for them, so I still don't understand what they were pointing at, but hey, it's not exactly my problem, so I ignored them. [edit] A friend told me that he suspects that this company might only use the abuse@ email address for police requests. If so, the response kinda makes sense, but still, they could have read what I mailed them. I suspect that their servers are still compromised, but that's up to them.
• Other companies also requested me to fill in a web form, giving me a username and password to do so and in the form ordering me to be very specific in stating the issue and delivering the proof. Having send emails all night I couldn't be bothered to toy around, so like the previous not, I suspect that their systems are still compromised as well.

Some larger hosting companies also asked me for logging, but with some of them having hundreds of compromised servers I asked them to give me some time to write a script to collect and combine all of their data. They replied they understood and after I had finished answering most of the emails I started working on that specific script.
I only included up to 4 lines of logging for each IP address and that still added up to combined log files of over 1.5MB for some of them.
So, I emailed the large files and all but one thanked me for the effort.

very, very, very weird :)

One of them though was not so sure and complained that the logging was not in their preferred format and they ordered me to change it according to their rules.
My suggestion to pay me for it never got an answer back. I do wonder why.

In the end I answered over 300 emails, had I realised this before I probably wouldn't have bothered. Serves me right for being both ignorant and helpful. :)
Most people appreciated the work I did, so I'll probably would do it again if it came to that. Still a sucker for appreciation. :(

Conclusion time;

do's;

• You really should need to have an abuse mailbox if you want people to be able to email you because you have compromised servers
• When you receive a message, understand that someone is doing you a favour. Appreciate that in itself, even though it means they ruined your day.

  don't's;

• You shouldn't bother people with your policies, terms or bureaucratic issues. You cannot expect people to fill in forms, call hotlines or whatever you fancy.
• Expect anything else from them

If you want people to send you information about issues that you are somehow responsible for, make it as easy as possible for them to do so. Do not expect them to follow your procedures. They were doing you a favour, just appreciate that and accept that it was your company letting them down, even if you aren't to blame in the first place.


Notes;

• I'm not going to blame and shame anyone, so don't bother asking
• proving that I don't know anything about security by hacking me, my network of my facebook will prove nothing, because..
• I am not a security expert, nor planning to become one.
• The compromised servers were all WordPress instances, I know nothing about, so don't ask for tips, except..
• read this; http://blog.sucuri.net/2014/03/more-than-162000-wordpress-sites-used-for-distributed-denial-of-service-attack.html
• block XMLRPC, whatever that means. :)