Yesterday’s non-issue, todays record breaker…

How open memcached quickly escalated to a record breaking DDoS vehicle.

The Great Wave Off Kanagawa by Katsushika Hokusai recreated by Lego artist Nathan Sawaya a CC NC SA image by Mary Harrsch

In my previous column I already described the amplification phenomenon that is used in modern DDoS attacks to turn a small traffic stream into a large one. In short, the criminals that launch DDoS attacks send small requests to publicly exposed services and spoof the originating IP address. These services then reply to the spoofed source address with a reply that is much larger then the original request, therefor amplifying the attack traffic.

The table below shows a list of services known to be used for amplification attacks and their amplification factor.

Taken from https://www.us-cert.gov/ncas/alerts/TA14-017A

While some of these services have a legitimate reason to be publicly exposed (e.g. DNS, NTP, BitTorrent and to a lesser extend LDAP and TFTP) most services on this list should generally not be exposed to public internet.

A very recent addition to this list is memcached, a network service “intended for use in speeding up dynamic web applications by alleviating database load.” The open memcached problem has been around for a long time, the ShadowServer project has been scanning for it since at least January 2015.

Until about two weeks ago the internet seemed to generally ignore the memcached problem. Why? So far, the biggest problem with memcached was its total lack of authentication and authorisation. So, if all you stored in memcached was public information, then why should you bother about locking down access to memcached? If you are not using memcached at all, but just have it listening, why should you bother closing it down.

In the past years the number of publicly accessible memcached instances therefore did not change much.

# of open memcached instances over multiple years, from https://memcachedscan.shadowserver.org/stats/

This all changed very recently, when somebody discovered that memcached also listens to UDP packets and can then be used as an amplifier. And it is a big amplifier. For each byte sent to an open memcached, the answer is 10,000 to 51,000 times bigger then the question. Within two days after the first attack using this technique was observed, GitHub was hit by the biggest DDoS attack recorded sofar, a whopping 1.3 Terrabits per second (the amount that can be contained on a 1TB harddisk every 6.5 seconds).

A sound I hear a CC NC image from mr_blage

There is a well known saying in information security that today’s incident is input for tomorrow’s policy and that’s demonstrably true for memcached do.

Providers like LeaseWeb are now actively blocking memcached traffic into their network.

Leaseweb’s notice about blocking memcached UDP traffic

And on a global scale the number of open memcached instances has since seen a dramatic drop.

# of open memcached instances over the last 3 months, from https://memcachedscan.shadowserver.org/stats/

Attacks of this size make a few things clear:

• Local appliances to mitigate DDoS attacks are pointless unless you (and your provider) have a tremendous line capacity
• There clearly is a rat-race going on between attackers and large DDoS mitigation providers
• Collective sloppiness/bad hygiene is hurting all of us
• There’s value in closing the stable door, even after the horse has bolted.

P.S. When I zoomed in on our own little corner of the world (.nl) I noticed something strange in the open mamcached numbers.

# of open memcached instances in The Netherlands over the last 3 months, from https://memcachedscan.shadowserver.org/stats/memcached_nl.html

All of a sudden there’s a sharp peak in the number open memcached instances in The Netherlands. Could it be that criminals are now installing memcached on compromised systems just to aid in DDoS attacks?