Broadband Developments

I just ran into this podcasts from the two gurus of DNS - Matt Larson and Cricket Liu.

For all you DNS junkies you’ll love this content from two old school DNS players.

Ask Mr. DNS Podcast

John Markoff’s New York Times recent story on the DNS exploit will no doubt draw significant attention to what Cricket Liu called one of the most significant vulnerabilities of all time. A few days after the easy to launch exploit was published on the Internet, evidence of attacks were soon reported, even against security experts including HD Moore, who was apparently also victimized by vulnerable AT&T servers.

This problem is particularly troubling because this flaw is widely known and present in an estimated 11 million servers responsible for directing traffic throughout the Internet. Kaminsky showed how the flaw could be exploited in seconds, in effect revolutionizing the economics of identity theft.

While service providers have been patching the vulnerability with limited success, leaving millions of core servers exposed, the story gets worse. Recent news suggests that firewalls may have been impacted, including those widely deployed to protect servers. Compatibility issues between the DNS vulnerability patch and firewalls have been reported to create additional availability risks, which mean that patching could proceed even more slowly than before. Fixes are on the way.

This is clearly a fluid, dynamic situation and possibly a sign of the times as the Internet comes of age.

While news of vulnerabilities, exploits and the sheer magnitude of this problem spreads, perhaps there is a silver lining. Perhaps CIOs will start dealing with the core challenge inadvertently laid bare by Kaminsky: that the Internet has outgrown its caretakers.

A Historical Perspective

In the early 1990s the Internet quickly encircled the globe, and was soon transporting incomprehensible levels of traffic to mushrooming populations of endpoints. All the while we heard about how resilient the Internet was, because it was architected to survive a nuclear blast. After all, the nuclear blast was and still is the classic metaphor for total destruction. Yet no one ever considered the destructive power of an attack on the core of the Internet: integrity.

From an economic standpoint, the Kaminsky DNS exploit may be the Internet’s equivalent of a nuclear strike; yet it doesn’t require a PhD with years of training, specialized uranium enrichment equipment or even a sophisticated form of delivery. It can be launched in seconds by any one of tens of thousands of hackers from almost anywhere in the world.

A successful DNS exploit wouldn’t destroy the physical Internet per se, but would rather neutralize its core integrity, its ability to act as an ecommerce enabler. Security and availability are, after all, the Internet’s bricks and mortar.

The Core Challenge

As the Internet exploded onto the scene it became responsible for transporting more traffic to more locations between more applications. Managing the domain names and addresses for a mushrooming population of endpoints created a market for more than 11 million DNS servers solely responsible for directing that traffic.

Not only are many of those servers past their prime, the methods for managing them have simply not kept up with their increasingly strategic importance. Hence patching the DNS vulnerability won’t be accomplished in a timely manner for many critical servers, even though the patch is the only protection and it still isn’t a permanent fix.

The core challenge to the success of the Internet going forward from the “Kaminsky event” isn’t really about applying a single patch, although the DNS vulnerability is probably the most significant security threat to the Internet since its inception. The core challenge will be related to how easily this large population of core servers can be managed, secured, updated and tracked.

In essence, the meteor has landed again in the world of technology, and flexibility and control will come to the forefront as a requirement for IT survival.

If an unprecedented vulnerability only gets patched on 1/3 of name servers after 30 days of industry headlines and relentless warnings from security experts; just how well managed will be other critical aspects of Internet integrity? Is anyone naïve enough to think that this will be the last threatening exploit against a list of known vulnerabilities or even zero day attacks (against undiscovered vulnerabilities)?

Kaminsky has indirectly proven that the caretakers of the Internet are today wholly incapable of protecting it. And the widely deployed tools and technologies once depended on are no longer sufficient for keeping up with the mushrooming role, complexity and demands of ensuring the integrity of the Internet.

The Rise of Core Network Services

This recent cache poisoning exploit event is likely to be one of many, and even the patch isn’t a permanent fix. The only long term solution, therefore, will require the automation of core network services and the proliferation of grid computing capabilities throughout public and private networks populated with DNS servers.

Core network services must move from being a scattered, freeware and spreadsheet dominated role to an advanced, strategic function supported by a new generation of dedicated appliances that automate critical functions and ensure proper reporting, accuracy and delegation of duties in seconds instead of days or weeks.

Kaminsky may have exposed a critical vulnerability in the Internet; he may also have become a catalyst for a more secure, more available and more robust Internet. While the New York Times featured the DNS challenge and Kaminsky, it has made it obvious that the solution is far bigger than any single patch or personality. It has heralded a new age in core network services.

You can access technical DNS resources at Cricket Liu’s DNS Resources Page or at DNSstuff.

You can read my disclaimer at: About « ARCHIMEDIUS.

Steve Ballmer gets it. While he discusses a strategic interest in search, his head is really in the clouds and beyond (hello new operating system models); in the coming transformation many are calling cloud computing. I think he fully understands the cannibalization risk that Google is posing in the long term as it delivers increasingly sophisticated applications as a service.

Yet there is another storm now appearing on the horizon for cloud computing, in addition to some technology challenges facing the proliferation of virtualization in the data center. Collectively they represent substantial, multifaceted risks to the major technology players.

While the media buzz surrounding Google, Yahoo, VMware and Microsoft has been particularly deafening this summer -between exec changes and various staged media events- the real story beneath the headlines is about a long term positioning battle being played out today between Microsoft and a new generation of upstarts over the delivery of software and how it’s monetized.

The VMware versus Microsoft battle is really a precursor to the coming cloud computing dogfight between Microsoft and Google, because virtualization is a critical enabler of cloud computing. And cloud computing will make certain technologies and capabilities strategic in ways that weren’t possible when data centers were cumbersome and inflexible.

Hypervisor Economics 101

The hypervisor revolution ignited by VMware enables new levels of flexibility and efficiency for managing even the most complex data center infrastructure, with point and click server management and movement. Multiple virtual machines (servers) can share the same hardware, regardless of operating system and be easily moved from one hypervisor to the next.

That new level of flexibility can transform the economics of IT, by delivering servers and processing power on an as-needed basis, versus keeping all hardware powered on even if only for potential use. Yet electricity savings are only part of the value proposition.

By converting broad collections of servers running different dedicated operating systems into sets of VMs running on larger blade servers, IT departments can make changes with minimal effort and their racks and stacks can take up a fraction of the space as was previously required. That could mean major transformations for service providers and large enterprises delivering applications to growing sets of users and partners.

Reducing power consumption and increasing agility could set the stage for a substantial shift to cloud computing. Yet hurdles remain. It is likely that virtualization security concerns have played a factor in VMware’s recent lackluster execution in the data center in 2008. Virtualization security is one of the major hurdles to virtualization and cloud computing.

Virtualization Security

I’ve called the nature of many virtualized production deployments virtualization-lite, because data centers accept a lower payoff from virtualization (less flexibility, less consolidation, reduced savings on electricity, for example) in exchange for maintaining their security posture. Players like Blue Lane Technologies (my alma mater) and others will be among the first to see the transformation of the data center as they are capable of protecting fluid meshes of hypervisors, a limitation for many types of network security appliances. That limitation has boxed in many virtualization projects into hypervisor VLANs, which substantially erode the business case.

Two Promising I/O Front Ends

Moving VMs around across hardware can also tie up additional processing overhead, which makes VMotion less than ideal at this time. Companies like 3 Leaf Systems and Xsigo Systems are addressing these challenges. As they grow they’ll be yet another proof point of the expansion of virtualization beyond hypervisor-VLANS, as their products enable greater flexibility.

There are also compliance and change management issues that might slow virtualization down and inadvertently buy Microsoft enough time to establish an even larger foothold in the data center market. VMware has been very effective in leveraging its partner ecosystem in addressing these issues.

Yet cloud computing faces a fair share or risks, including the biggest security story of perhaps the last ten years: the Kaminsky DNS exploit.

The New Storm Cloud for Cloud Computing

The last few weeks have seen a massive explosion in commentary on the DNS exploit discovered by security researcher Dan Kaminsky, Director of Penetration Testing at IOActive. Since his discovery and an inadvertent series of blog posts DNS cache poisoning exploit attack code has been published; and yesterday a ZDnet blog by security expert Dancho Danchev sited DNS cache poisoning attempts reported from multiple sources. Recent research also notes that a majority of service providers have not patched their systems for the vulnerability.

Infoblox Vice President Cricket Liu, the author of DNS and Bind, called it one of the most significant vulnerabilities of all time. Ironically, he was on a DNS Security: Old Vulnerabilities, New Exploits webinar with Dan Kaminsky just days before the exploit code was published.

The DNS exploit threatens the core integrity of the Internet, as it allows hackers to redirect traffic from exploited servers to spoof sites where they can gather personal information and engage in identity theft on a scale we have yet to experience. That’s a bigger problem than when the “I Love You” virus inconvenienced computer users years ago; it is a major storm front for the future of cloud computing.

An untrusted Internet would be nothing short of an ecommerce disaster; its impact would go far beyond cloud computing. It would be a major disruption for the software as a service model, as well as many other business models that have grown with the Internet. That’s why I predict that core network services will become increasingly strategic to IT. The integrity of the network is about to matter even more than ever.

As reported previously at Archimedius, Google and others have made considerable strides in delivering software as a service. Their success could mean the eventual shrinking of the computer hard drive, the shrinking of the pre-installed software market, not to mention the shrinking of the shrink-wrapped software industry.

Microsoft seems to understand the risks and upside, and has focused on “search” as a strategic roadmap issue, along with their recent Hyper-V attack on VMware. Yet the real Microsoft adversary is Google-driven cloud computing, and the spoiler issue for all of them is an untrusted Internet. Until a few months ago, few saw this issue coming. But now the vulnerability is known, exploits have been published and apparently attacks are now being launched.

You will be hearing much more about these issues, players and risks in coming weeks and probably months as Google and Microsoft prepare for battle in the skies.

You can read my disclaimer at: About ARCHIMEDIUS.

Based on a recent CERT Report published today at least 2/3 of Austrian recursive name servers have not yet been patched.

The conclusions are rather grim so far – more than two thirds of the Austrian Internet’s recursive

DNS servers are unpatched while at the same time the upgrade adoption rate seems rather slow.

Our findings are matched by the observations of Alexander Klink of Cynops GmbH2 who analyzed

the results of the online vulnerability test on Dan Kaminsky’s doxpara3 site.

- From Patching Nameservers: Austria Reacts to VU#800113

By Otmar Lendl and L. Aaron Kaplan July 24, 2008

It looks like Austria is NOT an anomaly but is rather symptomatic of many other countries behind on patching the DNS vulnerability and now exposed by the release of attack code. As the paper notes further on page 13, Alexander Klink had similar findings for doxpara.com queries.

Despite multiple warnings and the publication of exploit code it looks like successful attacks on the Internet are eminent.

From Cricket Liu’s exclusive Archimedius interview:

DNS experts agree that this vulnerability provides a way for a hacker to poison the cache of an unpatched, open recursive name server in less than a minute.

You can read my disclaimer at: About « ARCHIMEDIUS.

A few hours ago Ryan Naraine at ZDnet reported that DNS vulnerability attack code has been published.

The urgency to patch Dan Kaminsky’s DNS cache poisoning vulnerability just went up a few notches.

Exploit code for the flaw, which allows the insertion of malicious DNS records into the cache of the target nameserver, has been added to Metasploit, a freely distributed attack/pen-testing tool.

According to Metasploit creator HD Moore (left), who teamed up with researcher |)ruid to create the exploit, a DNS service has also been created to assist with the exploit.

Ryan Naraine, ZDnet July 23 2:55PM

You can read Cricket Liu’s breaking interview published this AM or attend a prescient webinar featuring Liu and Dan Kaminsky that was recorded just days ago. The original Archimedius coverage from July 22 contains links to various trade articles and resources.

You can read my disclaimer at: About « ARCHIMEDIUS.

Earlier this week the blogosphere and the press exploded with news about the inadvertent release of an exploit targeting a widely acknowledged vulnerability in about more than 11 million DNS servers. These servers are critical to the security of the Internet, as I mentioned yesterday at: DNS VULNERABILITY NOW IN THE WILD.

I found out about the release yesterday from Cricket Liu, the author of the definitive book on DNS, called DNS and BIND (published by O’Reilly). Cricket was on a DNS Security webcast with Dan Kaminsky a few days ago, and had then just spoken with Dan about the inadvertent release of the DNS vulnerability along with a researcher’s discovery of how an exploit could be successfully launched.

This of course puts extra pressures on administrators to patch their own DNS servers. If they don’t patch they expose users to cache poisoning attacks capable of redirecting them to spoof sites designed to collect personal information. This vulnerability, now in the wild, could turn the Internet into a hacker’s gold mine of passwords, account numbers and other identity theft resources.

Dan had planned to announce his findings, (discovered six months ago) at an upcoming (August) Black Hat conference, allowing administrators around the world adequate time to patch their DNS servers ahead of his presentation. Since the cat is now out of the bag according to Wired and other sources.

I decided to ask Cricket to get his take:

July 22, 2008 Interview with Cricket Liu

Q: If you were to rank Kaminsky’s recently disclosed DNS vulnerability, how would you rank it?

I assume you’re asking me to rank it among other DNS vulnerabilities. It’s certainly Number 1 today. It’s probably the All-Time Number 1, too, since we’ve always had solutions to address other DNS vulnerabilities. With this one, we have new versions of name servers that make the attacks more difficult to carry out, but no outright solution that’s been agreed on as of yet.

Q) How does it compare to other known vulnerabilities in terms of scope and potential impact and ease of exploit?

Well, the Kashpureff attack, back in July 1997, was easier to exploit. Name servers lacked mechanisms to detect unrelated additional data then, and almost all were open to recursive queries, so Kashpureff really had his pick of targets and could poison their caches almost instantly. It’s fortunate that he did so only to protest unfair business practices, not for his own gain. We didn’t see another exploit of that particular implementation flaw before implementations were fixed and name servers upgraded.

The current vulnerability is much broader in scope. There are many more name servers on the Internet today than there were in 1997, of course. Odds are the vulnerability is now widely known among the hacker community, after being revealed in a couple of security blogs yesterday. And if the anecdotal evidence I’m hearing is correct, many administrators aren’t upgrading their name servers to patched versions.

Q: For anyone who says that this latest DNS vulnerability is “business as usual” what would you tell them?

To dust off their resumes.

Seriously, this is a Big Deal. DNS experts agree that this vulnerability provides a way for a hacker to poison the cache of an unpatched, open recursive name server in less than a minute. Dan Kaminsky did everything he could to buy us time to patch our name servers. The Internet Systems Consortium and a whole lotta vendors—including Infoblox—worked hard to make sure you had patched code available the day of Dan’s announcement. If you stick your head in the sand and ignore the warnings, and a hacker writes code that combs the Internet for vulnerable, open recursive name servers, poisoning the A record for windowsupdate.microsoft.com, say, and you end up with legions of pwned PCs, guess who’ll get the blame.

Q: What is the nature of this vulnerability that makes it noteworthy compared to previous vulnerability and patch announcements?

It’s notable because there are so many hosts affected (from our surveys with The Measurement Factory, there are about 11 million name servers on the Internet) and because the consequences of a successful compromise are so high. If your name server’s cache is poisoned, you could find (but might never notice) that all of your mail to a business partner is re-routed through a mail server-in-the-middle, where it’s copied for later perusal and then sent on to unwitting recipients. Your traffic to critical web sites could be intercepted, and login names, passwords, and credit card numbers sniffed and recorded.

Q: Why do you think some security pros don’t find such a significant vulnerability alarming?

Some aspects of the vulnerability are familiar. We’ve known about attacks involving additional data since 1997. We’ve known the message ID in DNS messages isn’t long enough for a long time, too. But it’s not the components of the attack that are important. It’s that you can assemble them into a very effective attack against recursive name servers. Or a killer robot—your choice.

Q: Why do you think that a number of administrators are hesitating to patch their DNS systems?

Well, it can be a lot of work if you’re running plain vanilla BIND name servers on UNIX or Linux. And Amit Klein of Trusteer found a flaw in the implementation of BIND’s pseudo-random number generator (used to generate message IDs) last year. Some administrators may think that the patches they applied for that vulnerability will protect them from this one. (They won’t.)

NOTE: Cricket also has a DNS Best Practices micro-site at www.infoblox.com.

You can read my disclaimer at: About « ARCHIMEDIUS.

There are about 11 million servers using the Internet’s Domain Name System (DNS) to coordinate traffic across the Internet to their proper destinations. About 6 months ago Dan Kaminsky, Director of penetration testing at IOActive, discovered a way to exploit long-known DNS vulnerabilities to easily implement “cache poisoning” attacks that can compromise the integrity of the Internet. A few highlights from Computerworld’s coverage of the DNS flaw follow:

“DNS servers are responsible for routing all Internet traffic to their correct destinations. The so-called cache-poisoning vulnerability that Kaminsky discovered could allow attackers to redirect Web traffic and e-mails to systems under their control, according security researches. The flaw exists at the DNS protocol level and affects numerous products from multiple vendors.”

Jaikumar Vijayan, Computerworld, July 17

Word of the DNS flaw was made public earlier this month thanks to a collaborative update from the likes of Cisco and Microsoft. Details were withheld in order to give administrators time to patch their systems.

The flaw would allow hackers to launch unlimited queries against DNS servers without being detected, allowing them to run simple random number guesses to collect transaction IDs and other critical information that could be used to redirect web traffic to spoof sites. These kinds of attacks can be successful, and in turn, detrimental to an organization’s web presence, in mere seconds.

“According to Kaminsky, a weakness exists in a transaction identification process that the DNS protocol uses to determine whether responses to DNS queries are legitimate or not. DNS messages include what are supposed to be random identification numbers, but the problem, according to Kaminsky, is that only about 65,000 different values are currently being used as identifiers. And in reality, the process of assigning the identifiers to packets isn’t especially random and can be guessed, he said.”

Jaikumar Vijayan, Computerworld, July 17

While some have speculated whether or not the vulnerability is old news, Mike Fratto had recently delivered a stern warning to patch all DNS servers in his InformationWeek blog:

“Since the CERT announcement yesterday about the new vulnerabilities in DNS, there has been a lot of speculation that what Dan Kaminsky found is old news. Thomas Ptacek from Matasano, in an interview with Nathan McFeters at ZDNet, pretty much dismisses the vulnerability as old news and therefore unimportant. That sentiment is echoed on mailing lists and message boards. But in an e-mail today, Kaminsky confirmed that what he found is something very new. I believe him. Forget the arguments. Go patch your DNS servers. Now.”

Mike Fratto, InformationWeek, July 9

Making matters worse, a slip-up between security researchers discussing the cache poisoning attack via blog exchanges has inadvertently released details of how to launch an exploit in the wild, making it only a matter of time before real attacks appear.

Here is the coverage from ZDnet yesterday afternoon: Has Halvar figured out super-secret DNS vulnerability?

Clearly irked by a demand request from Kaminsky and others to avoid speculating on the details of the flaw until the patch is fully deployed, Flake (left) published a reliable method to forge and poison DNS lookups.

Ryan Naraine, ZDnet, July 21

You can expect to read much more about this in the coming days, if not hours.

You can find out even more from this recent webinar hosted by Dan Kaminsky and Infoblox VP of Architecture Cricket Liu: DNS Security: Old Vulnerabilities, New Exploits. It is sponsored by Infoblox, and is perhaps one of the most current and informative recorded events on the topic. You can also read more at Kaminsky to discuss DNS flaw at Black Hat sponsored webcast.

For more background, you can read the following articles:

internetnews.com: Who is Really at Risk From the DNS Flaw?

internetnews.com: Is DNSSEC the Answer to Internet Security?

InformationWeek blog: Stop Arguing and Patch your DNS

Computerworld: DNS flaw discoverer says more permanent fixes will be needed

You can read my disclosure at: About Archimedius .

There are about 11 million servers using the Internet’s core Domain Name System (DNS) protocol to coordinate traffic across the Internet to their proper destinations. About 6 months ago Dan Kaminsky, director of penetration testing at IOActive, discovered a way to exploit long-known DNS vulnerabilities to easily implement “cache poisoning” attacks that can compromise the integrity of the Internet.

A few highlights from Computerworld’s coverage of the DNS flaw follow:

DNS servers are responsible for routing all Internet traffic to their correct destinations. The so-called cache-poisoning vulnerability that Kaminsky discovered could allow attackers to redirect Web traffic and e-mails to systems under their control, according security researches. The flaw exists at the DNS protocol level and affects numerous products from multiple vendors.

Jaikumar Vijayan, Computerworld, July 17

Word of the DNS flaw was made public earlier this month thanks to a collaborative update from the likes of Cisco and Microsoft. Hackers could launch unlimited queries against DNS servers without being detected, allowing them to run simple random number guesses to collect transaction IDs and other critical information that could be used to redirect web traffic to spoof sites.

These kinds of attacks can be successful, and in turn detrimental to an organization’s web presence, in mere seconds.

According to Kaminsky, a weakness exists in a transaction identification process that the DNS protocol uses to determine whether responses to DNS queries are legitimate or not. DNS messages include what are supposed to be random identification numbers, but the problem, according to Kaminsky, is that only about 65,000 different values are currently being used as identifiers. And in reality, the process of assigning the identifiers to packets isn’t especially random and can be guessed, he said.

Jaikumar Vijayan, Computerworld, July 17

While some have speculated whether or not the vulnerability is old news, Mike Fratto had recently delivered a stern warning to patch all DNS servers in his InformationWeek blog:

Since the CERT announcement yesterday about the new vulnerabilities in DNS, there has been a lot of speculation that what Dan Kaminsky found is old news. Thomas Ptacek from Matasano, in an interview with Nathan McFeters at ZDNet, pretty much dismisses the vulnerability as old news and therefore unimportant. That sentiment is echoed on mailing lists and message boards. But in an e-mail today, Kaminsky confirmed that what he found is something very new. I believe him. Forget the arguments. Go patch your DNS servers. Now.

Mike Fratto, InformationWeek, July 9

Making matters worse, a slip-up between security researchers discussing the cache poisoning attack via blog exchanges has today inadvertently released details of how to launch an exploit in the wild, making it only a matter of time before real attacks appear.

You can expect to read much more about this in the coming days, if not hours.

You can find out more from this recent webinar hosted by Dan Kaminsky and Infoblox VP of Architecture Cricket Liu: DNS Security: Old Vulnerabilities, New Exploits. It is sponsored by Infoblox, and is perhaps one of the most current and informative recorded events on the topic. Ironically, today is my first day at Infoblox.

For more background, you can read the following articles:

internetnews.com: Who is Really at Risk From the DNS Flaw?

internetnews.com: Is DNSSEC the Answer to Internet Security?

InformationWeek blog: Stop Arguing and Patch your DNS

Computerworld: DNS flaw discoverer says more permanent fixes will be needed

HowStuffWorks.com: How Domain Name Servers Work

Wikipedia: DNS cache poisoning

+++++You can read my disclosure at: About Archimedius. +++++