Broadband Developments

The biggest threat to the promise of cloud computing to appear this summer wasn’t the failed trademark attempt by Dell, but rather brilliant research by a leading white hat security researcher. Dan Kaminsky discovered how a well-known and widespread vulnerability in DNS servers could be exploited in seconds and turn any one of millions of servers directing Internet traffic into a cybercrime gold mine in mere seconds.

Note: For those unfamiliar with cloud computing, or the delivery of software and other IT-related functionality as a service, you can read more at Archimedius. Some leading technology players involved or associated with cloud computing include: Google, Microsoft, Dell, VMware and Amazon.

As a result July and August saw unprecedented DNS media attention. Yet the discovery of a DNS exploit was only part of the story. Events soon unfolded that took the exploit from specialized security blogs (like Rational Survivability and Matasano, where the exploit leaked).

When the exploit inadvertently leaked (ahead of the disclosure timeline established to allow service providers ample time to patch their systems) the news quickly spread throughout more generalist blogs and even into mainstream media, including front page coverage in the NY Times referenced at Archimedius on July 31.

The Linux Journal published one of the best high level technical explanations of the exploit and why it matters. Despite the release of a patch and the heroic actions on the part of internet service providers, issues remain.

While the business press dwells on Dell, Microsoft, Google and a handful of key players making investments and strategic moves based on the eventuality of cloud computing, some of us in security and networking are all too aware of the storm clouds. You can read about the security issues at the newly established Infoblox DNS Security Center, with news, developments and resources hand-picked by leading experts.

Dan Kaminsky has openly labeled the patch just applied to protect the DNS vulnerability a temporary fix:

I listened to the Black Hat webcast today to grab as much info as I could on this subject. The biggest thing that I heard from the whole talk is that the patch fixes things to a reasonable point, but that long-term, there will have to be more work done to prevent the issue.

- Nathan McFeters, ZDNet

Unfortunately, it is likely that the DNS summer exploit story will fall back beneath the headlines in coming months; yet the vulnerability will still exist and it will likely require more patches on an ongoing basis. That will place an unprecedented level of demands on the management of the DNS infrastructure, the backbone of the Internet. That infrastructure is made up of millions of servers updated and managed manually. That is a serious problem.

An IDC report sponsored by Microsoft concluded that hardware costs were only a small fraction of the cost of operating a server (see page 5 for the IDC breakdown). Staffing expenses (management) and downtime constituted 75% of a server’s total cost of ownership, according to the April 2007 paper by Randy Perry and Al Gillen. More manual updates will impact both management and availability, the leading cost components before the DNS exploit discovery.

Internet integrity is a critical requirement for cloud computing. It requires a very high level of trust to use an online application for commercial and even personal uses. More management and availability challenges will further increase the cost of internet integrity while introducing new risks. The DNS exploit and the recognition that the recent patch is only a short term measure suggests that internet integrity may be more at risk than ever.

There’s More

A few days ago I discovered this YouTube piece by Cisco promoting green data centers and couldn’t help but to take notice of the points made about other server costs, including power. Cloud computing could suck up huge amounts of energy if cloudplexes are not virtualized properly and managed efficiently. For all of the opportunities posed by cloud computing it is obvious that substantial technical burdens remain before servers will follow the moon In pursuit of cheap electricity.

While low cost electricity and VMotion are important requirements for cloud computing, Internet integrity is the table stake: few will trust IT services from an unknown source. That is why the rise of cloud computing will depend upon the continued success and evolution of utility-grade core network services. Without network integrity the economics of software as a service will always be limited to low value consumers using low value services.

You can read my disclaimer at: About ARCHIMEDIUS.

The old saying if it bleeds it leads but this is getting out of hand. The DNS story is a real one, but lets move on to solutions - it’s clear that this as news is “beating this dead horse”. Enough of the gloom and doom. There is no doubt Dan Kaminsky is lovin the visability, but enough already on the problem. Time to move on to solutions.

Everyone knows DNS has a tons of holes but there are fixes and commercial software like Infoblox (now a sponsor of this blog - Thanks Infoblox).

Here is the timeline of this global conversation on BroadDev on the DNS story (with links externally to other credible sources): All of our contributors have chimed in on this topic.. very relevant.

July 22, 2008 - DNS Vunerability Has Now Gone Wild

July 23, 2008 - DNS Gone Wild - Exclusive Interview with Cricket Lui

July 24, 2008 - ZDNet Reports that DNS Exploit Code Has Been Published

July 24, 2008 - Cert: 60% of Recursive Name Servers UnPatched

July 25, 2008 - DNS Exploit is a Sleeping Zombie - Get the Patch

July 25, 2008 - Is Change Control Making the DNS Worse?

July 29, 2008 - DNS Exploit Again - It Keeps Going and Going - Feels like the Energizer Bunny of Exploits

July 29, 2008 - Breaking News: Now Patch Your Firewalls Because the DSN Patch Won’t Work With Leading Firewalls

July 30, 2008 - DNS SUCKS - Ok I Said It - Now What - Talk to Trusted Sources Until PAT mode is Fixed

July 31, 2008 - DNS Flaw Could Disrupt Unified Communications

July 31, 2008 - Kaminsky’s DNS Exploit Exposes Internet Core Challenge

August 5, 2008 - Black Hat 2008 - Look for Social Nets and DNS to Be Hot Topics

August 7, 2008 - Leaked Memo: DNS Security Flaw - Worst Security Hole Since 1997

These links are just the BroadDev coverage - This thing went supernova when John Markoff put it front and center in the NYTimes.

Time for solutions please - this as news is a dead horse.

John Markoff’s New York Times recent story on the DNS exploit will no doubt draw significant attention to what Cricket Liu called one of the most significant vulnerabilities of all time. A few days after the easy to launch exploit was published on the Internet, evidence of attacks were soon reported, even against security experts including HD Moore, who was apparently also victimized by vulnerable AT&T servers.

This problem is particularly troubling because this flaw is widely known and present in an estimated 11 million servers responsible for directing traffic throughout the Internet. Kaminsky showed how the flaw could be exploited in seconds, in effect revolutionizing the economics of identity theft.

While service providers have been patching the vulnerability with limited success, leaving millions of core servers exposed, the story gets worse. Recent news suggests that firewalls may have been impacted, including those widely deployed to protect servers. Compatibility issues between the DNS vulnerability patch and firewalls have been reported to create additional availability risks, which mean that patching could proceed even more slowly than before. Fixes are on the way.

This is clearly a fluid, dynamic situation and possibly a sign of the times as the Internet comes of age.

While news of vulnerabilities, exploits and the sheer magnitude of this problem spreads, perhaps there is a silver lining. Perhaps CIOs will start dealing with the core challenge inadvertently laid bare by Kaminsky: that the Internet has outgrown its caretakers.

A Historical Perspective

In the early 1990s the Internet quickly encircled the globe, and was soon transporting incomprehensible levels of traffic to mushrooming populations of endpoints. All the while we heard about how resilient the Internet was, because it was architected to survive a nuclear blast. After all, the nuclear blast was and still is the classic metaphor for total destruction. Yet no one ever considered the destructive power of an attack on the core of the Internet: integrity.

From an economic standpoint, the Kaminsky DNS exploit may be the Internet’s equivalent of a nuclear strike; yet it doesn’t require a PhD with years of training, specialized uranium enrichment equipment or even a sophisticated form of delivery. It can be launched in seconds by any one of tens of thousands of hackers from almost anywhere in the world.

A successful DNS exploit wouldn’t destroy the physical Internet per se, but would rather neutralize its core integrity, its ability to act as an ecommerce enabler. Security and availability are, after all, the Internet’s bricks and mortar.

The Core Challenge

As the Internet exploded onto the scene it became responsible for transporting more traffic to more locations between more applications. Managing the domain names and addresses for a mushrooming population of endpoints created a market for more than 11 million DNS servers solely responsible for directing that traffic.

Not only are many of those servers past their prime, the methods for managing them have simply not kept up with their increasingly strategic importance. Hence patching the DNS vulnerability won’t be accomplished in a timely manner for many critical servers, even though the patch is the only protection and it still isn’t a permanent fix.

The core challenge to the success of the Internet going forward from the “Kaminsky event” isn’t really about applying a single patch, although the DNS vulnerability is probably the most significant security threat to the Internet since its inception. The core challenge will be related to how easily this large population of core servers can be managed, secured, updated and tracked.

In essence, the meteor has landed again in the world of technology, and flexibility and control will come to the forefront as a requirement for IT survival.

If an unprecedented vulnerability only gets patched on 1/3 of name servers after 30 days of industry headlines and relentless warnings from security experts; just how well managed will be other critical aspects of Internet integrity? Is anyone naïve enough to think that this will be the last threatening exploit against a list of known vulnerabilities or even zero day attacks (against undiscovered vulnerabilities)?

Kaminsky has indirectly proven that the caretakers of the Internet are today wholly incapable of protecting it. And the widely deployed tools and technologies once depended on are no longer sufficient for keeping up with the mushrooming role, complexity and demands of ensuring the integrity of the Internet.

The Rise of Core Network Services

This recent cache poisoning exploit event is likely to be one of many, and even the patch isn’t a permanent fix. The only long term solution, therefore, will require the automation of core network services and the proliferation of grid computing capabilities throughout public and private networks populated with DNS servers.

Core network services must move from being a scattered, freeware and spreadsheet dominated role to an advanced, strategic function supported by a new generation of dedicated appliances that automate critical functions and ensure proper reporting, accuracy and delegation of duties in seconds instead of days or weeks.

Kaminsky may have exposed a critical vulnerability in the Internet; he may also have become a catalyst for a more secure, more available and more robust Internet. While the New York Times featured the DNS challenge and Kaminsky, it has made it obvious that the solution is far bigger than any single patch or personality. It has heralded a new age in core network services.

You can access technical DNS resources at Cricket Liu’s DNS Resources Page or at DNSstuff.

You can read my disclaimer at: About « ARCHIMEDIUS.

A new flaw has sharpened the debate over how to come up with a long-term solution to the broader problem of the lack of security in the Domain Name System, which was invented in 1983 and was not created with uses like online banking in mind or huge internetworked enterprises and service providers.

When you see John Markoff of the NYTimes explaining to normal people that there are DNS problems you know the suckiness of DNS has gone mainstream.

I blogged yesterday that Cisco firewalls were affected and rendered the DNS patch useless. Well that was true, BUT it’s not just Cisco - it’s everyone. There is a bigger picture. DNS sucks. There is too much legacy and critical infrastructure that is more important then some sort of url rewrite and a hacking of a 16 bit port translation (or PAT - Port Address Translation). It’s called ‘industrial strength’ software. Companies like Infoblox and Nominum have big businesses because they took the DNS technology and scaled it with security. Can DNS vendors do more with it or has it reached it’s peak? Either way this DNS shit is a big problem for IT and network operators. It seem like they are chasing too many holes out there. Is it time to rip and replace. I’ll keep my official opinion to myself.

Ok I’ll say it DNS sucks! This latest firewall PAT issue rendering the DNS patch useless is the latest example.

Richard Kagan of Infoblox chimed in this morning. Richard said “DNS is just a protocol. The challenges really tem form how it is administered. Companies haven’t historically treated DNS as a strategic asset and this recent vulnerability will likely focus a few more minds on DNS security, architecture, design, implementation and adminstration as well as the implications of past decisions.”

Firewall PAT Problem with DNS Patch

Regarding the firewall (and PAT devices), customers don’t have to really worry about this - just do the patch and get the upgrade from Cisco and others. The big deal is that there is a ton of critical infrastructure built ontop of the feeble DNS. We are talking about big businesses, big service provider networks, big data networks powering mobile devices, cable companies, etc .. all that rely on DNS.

Regarding the Cisco firewall problem - wait for the upgrade. The way Cisco firewalls allocate source ports and rewrite source ports in their PAT devices is sequential. Although this is an issue, it’s not a straightforward issue. There are many instances where multiple devices that rely on those ports need to run in legacy mode. Cisco told me today that they are releasing an option so that PAT can be configured to use a random number generator for their PAT mode devices. Some other disagree and say that there are more secure ways to go than with Cisco.

Depending on the implementation the firewall PAT problem can negate the DNS patch. Cisco will be changing their PAT mode and moving to “hardening of the PAT feature”. The upcoming configuration option will give customers the ability to make the PAT mode more random. The question will remain does this make the devices more secure? The PAT mode is 16 bit (very breakable). I’m waiting to hear.

I really like Cisco, but this has to be a huge pain in the ass for them (or anyone in IT networking). Is this a case of stupid DNS tricks or is this a bigger issue.

I’ll say it again DNS Sucks. This firewall PAT issue isn’t just a Cisco problem. Others are affected. In fact a story out of the UK today shows it’s also Checkpoint.

I am thankful that Cisco spent the time to talk to me. They were great and very candid and transparent. Maybe they could do a guest post to explain more. Or better yet get Ralph Droms (he and Cricket Lui wrote the book on DNS).

This DNS stuff is a mess. A patch will be released in a few weeks that will change the PAT from sequential to random.

The bigger picture is that DNS needs to be replaced. I can’t wait to have some experts talk with me more on this. It’s worth getting to the bottom of this issue.

Cisco says advises their customers to make sure that their devices only talks to a trusted source until the patch comes out in a few weeks”.

If you’re a Cisco customer then go to this link for DNS best practices for dealing with this issue.

Steve Ballmer gets it. While he discusses a strategic interest in search, his head is really in the clouds and beyond (hello new operating system models); in the coming transformation many are calling cloud computing. I think he fully understands the cannibalization risk that Google is posing in the long term as it delivers increasingly sophisticated applications as a service.

Yet there is another storm now appearing on the horizon for cloud computing, in addition to some technology challenges facing the proliferation of virtualization in the data center. Collectively they represent substantial, multifaceted risks to the major technology players.

While the media buzz surrounding Google, Yahoo, VMware and Microsoft has been particularly deafening this summer -between exec changes and various staged media events- the real story beneath the headlines is about a long term positioning battle being played out today between Microsoft and a new generation of upstarts over the delivery of software and how it’s monetized.

The VMware versus Microsoft battle is really a precursor to the coming cloud computing dogfight between Microsoft and Google, because virtualization is a critical enabler of cloud computing. And cloud computing will make certain technologies and capabilities strategic in ways that weren’t possible when data centers were cumbersome and inflexible.

Hypervisor Economics 101

The hypervisor revolution ignited by VMware enables new levels of flexibility and efficiency for managing even the most complex data center infrastructure, with point and click server management and movement. Multiple virtual machines (servers) can share the same hardware, regardless of operating system and be easily moved from one hypervisor to the next.

That new level of flexibility can transform the economics of IT, by delivering servers and processing power on an as-needed basis, versus keeping all hardware powered on even if only for potential use. Yet electricity savings are only part of the value proposition.

By converting broad collections of servers running different dedicated operating systems into sets of VMs running on larger blade servers, IT departments can make changes with minimal effort and their racks and stacks can take up a fraction of the space as was previously required. That could mean major transformations for service providers and large enterprises delivering applications to growing sets of users and partners.

Reducing power consumption and increasing agility could set the stage for a substantial shift to cloud computing. Yet hurdles remain. It is likely that virtualization security concerns have played a factor in VMware’s recent lackluster execution in the data center in 2008. Virtualization security is one of the major hurdles to virtualization and cloud computing.

Virtualization Security

I’ve called the nature of many virtualized production deployments virtualization-lite, because data centers accept a lower payoff from virtualization (less flexibility, less consolidation, reduced savings on electricity, for example) in exchange for maintaining their security posture. Players like Blue Lane Technologies (my alma mater) and others will be among the first to see the transformation of the data center as they are capable of protecting fluid meshes of hypervisors, a limitation for many types of network security appliances. That limitation has boxed in many virtualization projects into hypervisor VLANs, which substantially erode the business case.

Two Promising I/O Front Ends

Moving VMs around across hardware can also tie up additional processing overhead, which makes VMotion less than ideal at this time. Companies like 3 Leaf Systems and Xsigo Systems are addressing these challenges. As they grow they’ll be yet another proof point of the expansion of virtualization beyond hypervisor-VLANS, as their products enable greater flexibility.

There are also compliance and change management issues that might slow virtualization down and inadvertently buy Microsoft enough time to establish an even larger foothold in the data center market. VMware has been very effective in leveraging its partner ecosystem in addressing these issues.

Yet cloud computing faces a fair share or risks, including the biggest security story of perhaps the last ten years: the Kaminsky DNS exploit.

The New Storm Cloud for Cloud Computing

The last few weeks have seen a massive explosion in commentary on the DNS exploit discovered by security researcher Dan Kaminsky, Director of Penetration Testing at IOActive. Since his discovery and an inadvertent series of blog posts DNS cache poisoning exploit attack code has been published; and yesterday a ZDnet blog by security expert Dancho Danchev sited DNS cache poisoning attempts reported from multiple sources. Recent research also notes that a majority of service providers have not patched their systems for the vulnerability.

Infoblox Vice President Cricket Liu, the author of DNS and Bind, called it one of the most significant vulnerabilities of all time. Ironically, he was on a DNS Security: Old Vulnerabilities, New Exploits webinar with Dan Kaminsky just days before the exploit code was published.

The DNS exploit threatens the core integrity of the Internet, as it allows hackers to redirect traffic from exploited servers to spoof sites where they can gather personal information and engage in identity theft on a scale we have yet to experience. That’s a bigger problem than when the “I Love You” virus inconvenienced computer users years ago; it is a major storm front for the future of cloud computing.

An untrusted Internet would be nothing short of an ecommerce disaster; its impact would go far beyond cloud computing. It would be a major disruption for the software as a service model, as well as many other business models that have grown with the Internet. That’s why I predict that core network services will become increasingly strategic to IT. The integrity of the network is about to matter even more than ever.

As reported previously at Archimedius, Google and others have made considerable strides in delivering software as a service. Their success could mean the eventual shrinking of the computer hard drive, the shrinking of the pre-installed software market, not to mention the shrinking of the shrink-wrapped software industry.

Microsoft seems to understand the risks and upside, and has focused on “search” as a strategic roadmap issue, along with their recent Hyper-V attack on VMware. Yet the real Microsoft adversary is Google-driven cloud computing, and the spoiler issue for all of them is an untrusted Internet. Until a few months ago, few saw this issue coming. But now the vulnerability is known, exploits have been published and apparently attacks are now being launched.

You will be hearing much more about these issues, players and risks in coming weeks and probably months as Google and Microsoft prepare for battle in the skies.

You can read my disclaimer at: About ARCHIMEDIUS.

The exploit is still out there.  Apple Still has not patched the DNS vunerability.  This vunerability here has been running for weeks in the security circles.  It feels like the energizer bunny of vunerabilities.  People just get the damn patch done will you!  Enough already.  Ok- my rant is done.

On Slashdot Steve Shockley notes an article up at TidBITS on Apple’s unexplained failure to patch the DNS vulnerability that we have been discussing for a few weeks now. “Apple uses the popular Internet Systems Consortium BIND DNS server, which was one of the first tools patched, but Apple has yet to include the fixed version in Mac OS X Server, despite being notified of vulnerability details early in the process and being informed of the coordinated patch release date.”

More good stuff on Slashdot below:

Related posts from Slashdot

Kaminsky’s DNS Attack Disclosed, Then Pulled

A few hours ago Ryan Naraine at ZDnet reported that DNS vulnerability attack code has been published.

The urgency to patch Dan Kaminsky’s DNS cache poisoning vulnerability just went up a few notches.

Exploit code for the flaw, which allows the insertion of malicious DNS records into the cache of the target nameserver, has been added to Metasploit, a freely distributed attack/pen-testing tool.

According to Metasploit creator HD Moore (left), who teamed up with researcher |)ruid to create the exploit, a DNS service has also been created to assist with the exploit.

Ryan Naraine, ZDnet July 23 2:55PM

You can read Cricket Liu’s breaking interview published this AM or attend a prescient webinar featuring Liu and Dan Kaminsky that was recorded just days ago. The original Archimedius coverage from July 22 contains links to various trade articles and resources.

You can read my disclaimer at: About « ARCHIMEDIUS.