Blame the ISPs rather than the routers

The latest router security problem, that initially cropped up a week ago in Germany, has since been confirmed in other countries as well.

That its a new variant of Mirai, makes for sexy for headlines, but is not important. That five million devices may be vulnerable is also not important. And, It's pretty much irrelevant that the buggy routers were produced by Acadyan and Zyxel.

The most important issue in this latest router attack is that most of the blame falls on the Internet Service Providers (ISPs).

The routers were attacked on TCP port 7547, which is used by the TR-069 protocol (also known as CWMP or CPE WAN Management Protocol). Leaving port 7547 open would have been a non-issue if the ISPs had made better decisions.

A TOTALLY OPEN PORT

For example, it is common knowledge that open TCP/IP ports are dangerous. When an ISP wants to access their customer devices on an open port, they should take steps to insure that only they can do so. Instead, we saw ISPs mis-configuring both the routers and their internal networks such that any bad guy, anywhere in the world, can get into the routers on port 7547.

Its the network equivalent of not wearing seat belts. No surprise then, that customers were eventually thrown through the front window.

Security expert Graham Cluley was also annoyed about this, writing

The attacks ... are exploiting functionality which allows ISPs to remotely manage their customers' broadband routers. I can fully understand why ISPs want that kind of ability to reduce the support burden, but surely it would be better if connections were only allowed from the ISP's own managed network rather than any Tom, Dick or Harry based anywhere in the world?

Tod Beardsley and Bob Rudis of Rapid7 also griped about this:

For ISPs, it is A Bad Idea to expose either TR-069 or TR-064 services to arbitrary sources on the Internet; while ISPs may need access to this port to perform routine configuration maintenance on customer equipment, it should be possible for local and edge firewall rules to restrict access to this port to only IP addresses that originate from the ISP's management network.

The router bug that is being exploited was first discovered in early November by someone who goes by "kenzo2017." He found the flaw on a D1000 modem from Irish ISP Eir and chastised them writing:

Back in the days when Eir were Eircom and they used Netopia modems, port 7547 was blocked to every IP address except those assigned to Eir's management servers. This meant even though the Netopia modems had bugs, they could not be exploited. Inexplicably, Eir do not do this for their newer modems. If they did, these bugs would not have been exploitable.

Seat belts.

Not to pick on Eir, the same is true of Deutsche Telekom. Let's parse a couple sentences in their public statement.

... routers of Deutsche Telekom costumers were affected by an attack from outside.

In other words, their default network configuration allowed anyone in the world to hack into the routers of their customers, via an open port, and that is on them.

Their fix?

We implemented a series of filter measures to our network.

Network filters that should have been there all along.

Ironically, Deutsche Telekom brags that their Cyber Defense Center emphasizes "good preventive strategies."

Another effected ISP, TalkTalk, said that it too has "... deployed additional network-level controls to further protect our customers." UK ISP KCOM, likewise "... put in place measures to block future attacks from impacting our customers."

None of the ISPs took the obvious defensive step until they were shamed into it.

And, sadly, there is nothing special about these four ISPs. Reports say that Shodan found 41 million devices on the Internet with port 7547 open. If ISPs were limiting access to this port to just their servers that need it, then Shodan would not be able to detect that the port was open.

MIS-CONFIGURED ROUTERS

Another common ISP security mistake is giving their customers sub-optimal routers.

The routers that were hacked this week were buggy, which is not all that unusual.

The issue here seems to have been with the very old TR-064 protocol. According to a 2004 document from the DSLHome-Technical Working Group, the protocol is also known as "LAN-Side DSL CPE Configuration" (CPE is Customer Premises Equipment). The important point being that the protocol was designed to work on a LAN, that is, on the inside of your home. It was not designed or intended to be used over the Internet.

Yet, that's what the vulnerable routers were doing, they were responding to TR-064 commands sent to port 7547.

It's somewhat akin to a doctor operating on the wrong leg of a patient. How much competence does it take to keep protocols designed for LAN side use away from the Internet? Too much, obviously.

According to Dan Goodin, the Shodan search engine reports that "about five million [devices] expose TR-064 services to the outside world."

The same thing happened in 2013 with UPnP, another protocol designed for LAN use only. Back then, Rapid7 found 80 million devices that responded to UPnP over the Internet.

BUGGY ROUTERS

By the time a TR-064 server is exposed to the Internet at large, two mistakes have already been made. On top of this, there was a bug in the implementation of the TR-064 protocol.

As "Kenzo" pointed out, in the write-up that that started all this in early November, a flaw in the SetNTPServers function can be exploited to run commands on the device. This lets an attacker turn off the firewall and remotely administer the router. An attacker can get full control of vulnerable devices.

In addition, there is also an information disclosure issue. Bad guys can learn assorted information about the router, without exploiting the flaw, just by issuing some TR-064 commands. The specific information "Kenzo" cited included the serial number, firmware version, the Wi-Fi password, the SSID and the MAC address.

BAD ROUTERS

The bad guys in this case, had no intention of knocking people off the Internet. The malware was supposed to compromise the routers so they could be later used for income producing purposes.

What went wrong? It turns out, some routers stop working when asked to do too much actual routing.

As the malware tries to infect other routers, it creates network traffic. So much traffic, that it caused some routers to lock-up.

Johannes Ullrich, of The SANS Technology Institute, told Brian Krebs that the excessive network traffic crashed some routers that were not even vulnerable to the malware. These other routers simply could not handle all the incoming connection attempts on port 7547 from infected routers.

Security company Comsecuris did their own hacking of an Arcadyan router and confirmed that an excessive amount of incoming connections caused the router to hang. They wrote

... simply flooding the router with the above requests (TCP/UDP) did show however that the device first becomes much slower in accepting TCP connections on the external IP. Finally, it stopped accepting TCP connections on that interface altogether. This aligns with DTAG [Deutsche Telekom AG] customer reports that indicate that the issue temporarily went away by restarting the device. Due to the ongoing heavy Mirai botnet activity, the faulty behavior quickly reoccurred in practice. We conclude that this phenomena is what caused the outage for some DTAG customers this weekend ... various news outlets definitely released reports based on preliminary investigation information that was wrong.

Lorenzo Franceschi-Bicchierai of Motherboard claimed to be in contact with one of the attackers in this case. The bad guy apologized for the impact the malware had on a British ISP, but followed that up with "... they should give their customers better hardware ... too many requests freeze the shitty routers."

That's telling it like it is.

DEFENSIVE COMPUTING

The Defensive Computing lesson here is clear. ISPs can not be trusted with security.

This case showed them mis-configuring their internal networks, mis-configuring the routers they give to customers and choosing routers that fail under stress. Three strikes, you're out.

A fourth strike comes from Comsecuris which, in the course of their investigation, found other router vulnerabilities. Putting this in perspective, they wrote

This is not surprising based on the general state of end user router equipment, which is an ongoing pain point since years.

Speaking as someone who follows router security, this is not surprising to me either.

UPDATE: December 5, 2016. Added quote from Rapid7. Also added information on the software flaw and the information disclosure.

FEEDBACK

Now that Computerworld, and all of parent company IDG's websites, have eliminated user comments, you can still provide me feedback two ways. Privately, email me at my full name at Gmail. Public comments can be directed to me on twitter at @defensivecomput.

Michael Horowitz is an independent consultant who has long been focused on Defensive Computing.

Copyright © 2016 IDG Communications, Inc.