Broadband Developments

The biggest threat to the promise of cloud computing to appear this summer wasn’t the failed trademark attempt by Dell, but rather brilliant research by a leading white hat security researcher. Dan Kaminsky discovered how a well-known and widespread vulnerability in DNS servers could be exploited in seconds and turn any one of millions of servers directing Internet traffic into a cybercrime gold mine in mere seconds.

Note: For those unfamiliar with cloud computing, or the delivery of software and other IT-related functionality as a service, you can read more at Archimedius. Some leading technology players involved or associated with cloud computing include: Google, Microsoft, Dell, VMware and Amazon.

As a result July and August saw unprecedented DNS media attention. Yet the discovery of a DNS exploit was only part of the story. Events soon unfolded that took the exploit from specialized security blogs (like Rational Survivability and Matasano, where the exploit leaked).

When the exploit inadvertently leaked (ahead of the disclosure timeline established to allow service providers ample time to patch their systems) the news quickly spread throughout more generalist blogs and even into mainstream media, including front page coverage in the NY Times referenced at Archimedius on July 31.

The Linux Journal published one of the best high level technical explanations of the exploit and why it matters. Despite the release of a patch and the heroic actions on the part of internet service providers, issues remain.

While the business press dwells on Dell, Microsoft, Google and a handful of key players making investments and strategic moves based on the eventuality of cloud computing, some of us in security and networking are all too aware of the storm clouds. You can read about the security issues at the newly established Infoblox DNS Security Center, with news, developments and resources hand-picked by leading experts.

Dan Kaminsky has openly labeled the patch just applied to protect the DNS vulnerability a temporary fix:

I listened to the Black Hat webcast today to grab as much info as I could on this subject. The biggest thing that I heard from the whole talk is that the patch fixes things to a reasonable point, but that long-term, there will have to be more work done to prevent the issue.

- Nathan McFeters, ZDNet

Unfortunately, it is likely that the DNS summer exploit story will fall back beneath the headlines in coming months; yet the vulnerability will still exist and it will likely require more patches on an ongoing basis. That will place an unprecedented level of demands on the management of the DNS infrastructure, the backbone of the Internet. That infrastructure is made up of millions of servers updated and managed manually. That is a serious problem.

An IDC report sponsored by Microsoft concluded that hardware costs were only a small fraction of the cost of operating a server (see page 5 for the IDC breakdown). Staffing expenses (management) and downtime constituted 75% of a server’s total cost of ownership, according to the April 2007 paper by Randy Perry and Al Gillen. More manual updates will impact both management and availability, the leading cost components before the DNS exploit discovery.

Internet integrity is a critical requirement for cloud computing. It requires a very high level of trust to use an online application for commercial and even personal uses. More management and availability challenges will further increase the cost of internet integrity while introducing new risks. The DNS exploit and the recognition that the recent patch is only a short term measure suggests that internet integrity may be more at risk than ever.

There’s More

A few days ago I discovered this YouTube piece by Cisco promoting green data centers and couldn’t help but to take notice of the points made about other server costs, including power. Cloud computing could suck up huge amounts of energy if cloudplexes are not virtualized properly and managed efficiently. For all of the opportunities posed by cloud computing it is obvious that substantial technical burdens remain before servers will follow the moon In pursuit of cheap electricity.

While low cost electricity and VMotion are important requirements for cloud computing, Internet integrity is the table stake: few will trust IT services from an unknown source. That is why the rise of cloud computing will depend upon the continued success and evolution of utility-grade core network services. Without network integrity the economics of software as a service will always be limited to low value consumers using low value services.

You can read my disclaimer at: About ARCHIMEDIUS.

John Markoff’s New York Times recent story on the DNS exploit will no doubt draw significant attention to what Cricket Liu called one of the most significant vulnerabilities of all time. A few days after the easy to launch exploit was published on the Internet, evidence of attacks were soon reported, even against security experts including HD Moore, who was apparently also victimized by vulnerable AT&T servers.

This problem is particularly troubling because this flaw is widely known and present in an estimated 11 million servers responsible for directing traffic throughout the Internet. Kaminsky showed how the flaw could be exploited in seconds, in effect revolutionizing the economics of identity theft.

While service providers have been patching the vulnerability with limited success, leaving millions of core servers exposed, the story gets worse. Recent news suggests that firewalls may have been impacted, including those widely deployed to protect servers. Compatibility issues between the DNS vulnerability patch and firewalls have been reported to create additional availability risks, which mean that patching could proceed even more slowly than before. Fixes are on the way.

This is clearly a fluid, dynamic situation and possibly a sign of the times as the Internet comes of age.

While news of vulnerabilities, exploits and the sheer magnitude of this problem spreads, perhaps there is a silver lining. Perhaps CIOs will start dealing with the core challenge inadvertently laid bare by Kaminsky: that the Internet has outgrown its caretakers.

A Historical Perspective

In the early 1990s the Internet quickly encircled the globe, and was soon transporting incomprehensible levels of traffic to mushrooming populations of endpoints. All the while we heard about how resilient the Internet was, because it was architected to survive a nuclear blast. After all, the nuclear blast was and still is the classic metaphor for total destruction. Yet no one ever considered the destructive power of an attack on the core of the Internet: integrity.

From an economic standpoint, the Kaminsky DNS exploit may be the Internet’s equivalent of a nuclear strike; yet it doesn’t require a PhD with years of training, specialized uranium enrichment equipment or even a sophisticated form of delivery. It can be launched in seconds by any one of tens of thousands of hackers from almost anywhere in the world.

A successful DNS exploit wouldn’t destroy the physical Internet per se, but would rather neutralize its core integrity, its ability to act as an ecommerce enabler. Security and availability are, after all, the Internet’s bricks and mortar.

The Core Challenge

As the Internet exploded onto the scene it became responsible for transporting more traffic to more locations between more applications. Managing the domain names and addresses for a mushrooming population of endpoints created a market for more than 11 million DNS servers solely responsible for directing that traffic.

Not only are many of those servers past their prime, the methods for managing them have simply not kept up with their increasingly strategic importance. Hence patching the DNS vulnerability won’t be accomplished in a timely manner for many critical servers, even though the patch is the only protection and it still isn’t a permanent fix.

The core challenge to the success of the Internet going forward from the “Kaminsky event” isn’t really about applying a single patch, although the DNS vulnerability is probably the most significant security threat to the Internet since its inception. The core challenge will be related to how easily this large population of core servers can be managed, secured, updated and tracked.

In essence, the meteor has landed again in the world of technology, and flexibility and control will come to the forefront as a requirement for IT survival.

If an unprecedented vulnerability only gets patched on 1/3 of name servers after 30 days of industry headlines and relentless warnings from security experts; just how well managed will be other critical aspects of Internet integrity? Is anyone naïve enough to think that this will be the last threatening exploit against a list of known vulnerabilities or even zero day attacks (against undiscovered vulnerabilities)?

Kaminsky has indirectly proven that the caretakers of the Internet are today wholly incapable of protecting it. And the widely deployed tools and technologies once depended on are no longer sufficient for keeping up with the mushrooming role, complexity and demands of ensuring the integrity of the Internet.

The Rise of Core Network Services

This recent cache poisoning exploit event is likely to be one of many, and even the patch isn’t a permanent fix. The only long term solution, therefore, will require the automation of core network services and the proliferation of grid computing capabilities throughout public and private networks populated with DNS servers.

Core network services must move from being a scattered, freeware and spreadsheet dominated role to an advanced, strategic function supported by a new generation of dedicated appliances that automate critical functions and ensure proper reporting, accuracy and delegation of duties in seconds instead of days or weeks.

Kaminsky may have exposed a critical vulnerability in the Internet; he may also have become a catalyst for a more secure, more available and more robust Internet. While the New York Times featured the DNS challenge and Kaminsky, it has made it obvious that the solution is far bigger than any single patch or personality. It has heralded a new age in core network services.

You can access technical DNS resources at Cricket Liu’s DNS Resources Page or at DNSstuff.

You can read my disclaimer at: About « ARCHIMEDIUS.

Based on a recent CERT Report published today at least 2/3 of Austrian recursive name servers have not yet been patched.

The conclusions are rather grim so far – more than two thirds of the Austrian Internet’s recursive

DNS servers are unpatched while at the same time the upgrade adoption rate seems rather slow.

Our findings are matched by the observations of Alexander Klink of Cynops GmbH2 who analyzed

the results of the online vulnerability test on Dan Kaminsky’s doxpara3 site.

- From Patching Nameservers: Austria Reacts to VU#800113

By Otmar Lendl and L. Aaron Kaplan July 24, 2008

It looks like Austria is NOT an anomaly but is rather symptomatic of many other countries behind on patching the DNS vulnerability and now exposed by the release of attack code. As the paper notes further on page 13, Alexander Klink had similar findings for doxpara.com queries.

Despite multiple warnings and the publication of exploit code it looks like successful attacks on the Internet are eminent.

From Cricket Liu’s exclusive Archimedius interview:

DNS experts agree that this vulnerability provides a way for a hacker to poison the cache of an unpatched, open recursive name server in less than a minute.

You can read my disclaimer at: About « ARCHIMEDIUS.

A few hours ago Ryan Naraine at ZDnet reported that DNS vulnerability attack code has been published.

The urgency to patch Dan Kaminsky’s DNS cache poisoning vulnerability just went up a few notches.

Exploit code for the flaw, which allows the insertion of malicious DNS records into the cache of the target nameserver, has been added to Metasploit, a freely distributed attack/pen-testing tool.

According to Metasploit creator HD Moore (left), who teamed up with researcher |)ruid to create the exploit, a DNS service has also been created to assist with the exploit.

Ryan Naraine, ZDnet July 23 2:55PM

You can read Cricket Liu’s breaking interview published this AM or attend a prescient webinar featuring Liu and Dan Kaminsky that was recorded just days ago. The original Archimedius coverage from July 22 contains links to various trade articles and resources.

You can read my disclaimer at: About « ARCHIMEDIUS.

Earlier this week the blogosphere and the press exploded with news about the inadvertent release of an exploit targeting a widely acknowledged vulnerability in about more than 11 million DNS servers. These servers are critical to the security of the Internet, as I mentioned yesterday at: DNS VULNERABILITY NOW IN THE WILD.

I found out about the release yesterday from Cricket Liu, the author of the definitive book on DNS, called DNS and BIND (published by O’Reilly). Cricket was on a DNS Security webcast with Dan Kaminsky a few days ago, and had then just spoken with Dan about the inadvertent release of the DNS vulnerability along with a researcher’s discovery of how an exploit could be successfully launched.

This of course puts extra pressures on administrators to patch their own DNS servers. If they don’t patch they expose users to cache poisoning attacks capable of redirecting them to spoof sites designed to collect personal information. This vulnerability, now in the wild, could turn the Internet into a hacker’s gold mine of passwords, account numbers and other identity theft resources.

Dan had planned to announce his findings, (discovered six months ago) at an upcoming (August) Black Hat conference, allowing administrators around the world adequate time to patch their DNS servers ahead of his presentation. Since the cat is now out of the bag according to Wired and other sources.

I decided to ask Cricket to get his take:

July 22, 2008 Interview with Cricket Liu

Q: If you were to rank Kaminsky’s recently disclosed DNS vulnerability, how would you rank it?

I assume you’re asking me to rank it among other DNS vulnerabilities. It’s certainly Number 1 today. It’s probably the All-Time Number 1, too, since we’ve always had solutions to address other DNS vulnerabilities. With this one, we have new versions of name servers that make the attacks more difficult to carry out, but no outright solution that’s been agreed on as of yet.

Q) How does it compare to other known vulnerabilities in terms of scope and potential impact and ease of exploit?

Well, the Kashpureff attack, back in July 1997, was easier to exploit. Name servers lacked mechanisms to detect unrelated additional data then, and almost all were open to recursive queries, so Kashpureff really had his pick of targets and could poison their caches almost instantly. It’s fortunate that he did so only to protest unfair business practices, not for his own gain. We didn’t see another exploit of that particular implementation flaw before implementations were fixed and name servers upgraded.

The current vulnerability is much broader in scope. There are many more name servers on the Internet today than there were in 1997, of course. Odds are the vulnerability is now widely known among the hacker community, after being revealed in a couple of security blogs yesterday. And if the anecdotal evidence I’m hearing is correct, many administrators aren’t upgrading their name servers to patched versions.

Q: For anyone who says that this latest DNS vulnerability is “business as usual” what would you tell them?

To dust off their resumes.

Seriously, this is a Big Deal. DNS experts agree that this vulnerability provides a way for a hacker to poison the cache of an unpatched, open recursive name server in less than a minute. Dan Kaminsky did everything he could to buy us time to patch our name servers. The Internet Systems Consortium and a whole lotta vendors—including Infoblox—worked hard to make sure you had patched code available the day of Dan’s announcement. If you stick your head in the sand and ignore the warnings, and a hacker writes code that combs the Internet for vulnerable, open recursive name servers, poisoning the A record for windowsupdate.microsoft.com, say, and you end up with legions of pwned PCs, guess who’ll get the blame.

Q: What is the nature of this vulnerability that makes it noteworthy compared to previous vulnerability and patch announcements?

It’s notable because there are so many hosts affected (from our surveys with The Measurement Factory, there are about 11 million name servers on the Internet) and because the consequences of a successful compromise are so high. If your name server’s cache is poisoned, you could find (but might never notice) that all of your mail to a business partner is re-routed through a mail server-in-the-middle, where it’s copied for later perusal and then sent on to unwitting recipients. Your traffic to critical web sites could be intercepted, and login names, passwords, and credit card numbers sniffed and recorded.

Q: Why do you think some security pros don’t find such a significant vulnerability alarming?

Some aspects of the vulnerability are familiar. We’ve known about attacks involving additional data since 1997. We’ve known the message ID in DNS messages isn’t long enough for a long time, too. But it’s not the components of the attack that are important. It’s that you can assemble them into a very effective attack against recursive name servers. Or a killer robot—your choice.

Q: Why do you think that a number of administrators are hesitating to patch their DNS systems?

Well, it can be a lot of work if you’re running plain vanilla BIND name servers on UNIX or Linux. And Amit Klein of Trusteer found a flaw in the implementation of BIND’s pseudo-random number generator (used to generate message IDs) last year. Some administrators may think that the patches they applied for that vulnerability will protect them from this one. (They won’t.)

NOTE: Cricket also has a DNS Best Practices micro-site at www.infoblox.com.

You can read my disclaimer at: About « ARCHIMEDIUS.

There are about 11 million servers using the Internet’s Domain Name System (DNS) to coordinate traffic across the Internet to their proper destinations. About 6 months ago Dan Kaminsky, Director of penetration testing at IOActive, discovered a way to exploit long-known DNS vulnerabilities to easily implement “cache poisoning” attacks that can compromise the integrity of the Internet. A few highlights from Computerworld’s coverage of the DNS flaw follow:

“DNS servers are responsible for routing all Internet traffic to their correct destinations. The so-called cache-poisoning vulnerability that Kaminsky discovered could allow attackers to redirect Web traffic and e-mails to systems under their control, according security researches. The flaw exists at the DNS protocol level and affects numerous products from multiple vendors.”

Jaikumar Vijayan, Computerworld, July 17

Word of the DNS flaw was made public earlier this month thanks to a collaborative update from the likes of Cisco and Microsoft. Details were withheld in order to give administrators time to patch their systems.

The flaw would allow hackers to launch unlimited queries against DNS servers without being detected, allowing them to run simple random number guesses to collect transaction IDs and other critical information that could be used to redirect web traffic to spoof sites. These kinds of attacks can be successful, and in turn, detrimental to an organization’s web presence, in mere seconds.

“According to Kaminsky, a weakness exists in a transaction identification process that the DNS protocol uses to determine whether responses to DNS queries are legitimate or not. DNS messages include what are supposed to be random identification numbers, but the problem, according to Kaminsky, is that only about 65,000 different values are currently being used as identifiers. And in reality, the process of assigning the identifiers to packets isn’t especially random and can be guessed, he said.”

Jaikumar Vijayan, Computerworld, July 17

While some have speculated whether or not the vulnerability is old news, Mike Fratto had recently delivered a stern warning to patch all DNS servers in his InformationWeek blog:

“Since the CERT announcement yesterday about the new vulnerabilities in DNS, there has been a lot of speculation that what Dan Kaminsky found is old news. Thomas Ptacek from Matasano, in an interview with Nathan McFeters at ZDNet, pretty much dismisses the vulnerability as old news and therefore unimportant. That sentiment is echoed on mailing lists and message boards. But in an e-mail today, Kaminsky confirmed that what he found is something very new. I believe him. Forget the arguments. Go patch your DNS servers. Now.”

Mike Fratto, InformationWeek, July 9

Making matters worse, a slip-up between security researchers discussing the cache poisoning attack via blog exchanges has inadvertently released details of how to launch an exploit in the wild, making it only a matter of time before real attacks appear.

Here is the coverage from ZDnet yesterday afternoon: Has Halvar figured out super-secret DNS vulnerability?

Clearly irked by a demand request from Kaminsky and others to avoid speculating on the details of the flaw until the patch is fully deployed, Flake (left) published a reliable method to forge and poison DNS lookups.

Ryan Naraine, ZDnet, July 21

You can expect to read much more about this in the coming days, if not hours.

You can find out even more from this recent webinar hosted by Dan Kaminsky and Infoblox VP of Architecture Cricket Liu: DNS Security: Old Vulnerabilities, New Exploits. It is sponsored by Infoblox, and is perhaps one of the most current and informative recorded events on the topic. You can also read more at Kaminsky to discuss DNS flaw at Black Hat sponsored webcast.

For more background, you can read the following articles:

internetnews.com: Who is Really at Risk From the DNS Flaw?

internetnews.com: Is DNSSEC the Answer to Internet Security?

InformationWeek blog: Stop Arguing and Patch your DNS

Computerworld: DNS flaw discoverer says more permanent fixes will be needed

You can read my disclosure at: About Archimedius .

There are about 11 million servers using the Internet’s core Domain Name System (DNS) protocol to coordinate traffic across the Internet to their proper destinations. About 6 months ago Dan Kaminsky, director of penetration testing at IOActive, discovered a way to exploit long-known DNS vulnerabilities to easily implement “cache poisoning” attacks that can compromise the integrity of the Internet.

A few highlights from Computerworld’s coverage of the DNS flaw follow:

DNS servers are responsible for routing all Internet traffic to their correct destinations. The so-called cache-poisoning vulnerability that Kaminsky discovered could allow attackers to redirect Web traffic and e-mails to systems under their control, according security researches. The flaw exists at the DNS protocol level and affects numerous products from multiple vendors.

Jaikumar Vijayan, Computerworld, July 17

Word of the DNS flaw was made public earlier this month thanks to a collaborative update from the likes of Cisco and Microsoft. Hackers could launch unlimited queries against DNS servers without being detected, allowing them to run simple random number guesses to collect transaction IDs and other critical information that could be used to redirect web traffic to spoof sites.

These kinds of attacks can be successful, and in turn detrimental to an organization’s web presence, in mere seconds.

According to Kaminsky, a weakness exists in a transaction identification process that the DNS protocol uses to determine whether responses to DNS queries are legitimate or not. DNS messages include what are supposed to be random identification numbers, but the problem, according to Kaminsky, is that only about 65,000 different values are currently being used as identifiers. And in reality, the process of assigning the identifiers to packets isn’t especially random and can be guessed, he said.

Jaikumar Vijayan, Computerworld, July 17

While some have speculated whether or not the vulnerability is old news, Mike Fratto had recently delivered a stern warning to patch all DNS servers in his InformationWeek blog:

Since the CERT announcement yesterday about the new vulnerabilities in DNS, there has been a lot of speculation that what Dan Kaminsky found is old news. Thomas Ptacek from Matasano, in an interview with Nathan McFeters at ZDNet, pretty much dismisses the vulnerability as old news and therefore unimportant. That sentiment is echoed on mailing lists and message boards. But in an e-mail today, Kaminsky confirmed that what he found is something very new. I believe him. Forget the arguments. Go patch your DNS servers. Now.

Mike Fratto, InformationWeek, July 9

Making matters worse, a slip-up between security researchers discussing the cache poisoning attack via blog exchanges has today inadvertently released details of how to launch an exploit in the wild, making it only a matter of time before real attacks appear.

You can expect to read much more about this in the coming days, if not hours.

You can find out more from this recent webinar hosted by Dan Kaminsky and Infoblox VP of Architecture Cricket Liu: DNS Security: Old Vulnerabilities, New Exploits. It is sponsored by Infoblox, and is perhaps one of the most current and informative recorded events on the topic. Ironically, today is my first day at Infoblox.

For more background, you can read the following articles:

internetnews.com: Who is Really at Risk From the DNS Flaw?

internetnews.com: Is DNSSEC the Answer to Internet Security?

InformationWeek blog: Stop Arguing and Patch your DNS

Computerworld: DNS flaw discoverer says more permanent fixes will be needed

HowStuffWorks.com: How Domain Name Servers Work

Wikipedia: DNS cache poisoning

+++++You can read my disclosure at: About Archimedius. +++++