100 Gigabit Ethernet: Bridge to Terabit Ethernet

April 20, 2009 (Network World) IT managers who are getting started with -- or even pushing the limits of -- 10 Gigabit Ethernet in their LANs and data centers won't have to wait long for higher speed connectivity.

Podcast: Pushing Beyond 100 Gigabit Ethernet
Slideshow: The evolution of Ethernet

Pre-standard 40 Gigabit and 100 Gigabit Ethernet products -- server network interface cards, switch uplinks and switches -- are expected to hit the market later this year. And standards-compliant products are expected to ship in the second half of next year, not long after the expected June 2010 ratification of the 802.3ba standard.

The IEEE, which began work on the standard in late 2006, is expected to define two different speeds of Ethernet for two different applications: 40G for server connectivity and 100G for core switching.

Despite the global economic slowdown, global revenue for 10G fixed Ethernet switches doubled in 2008, according to Infonetics. And there is pent-up demand for 40 Gigabit and 100 Gigabit Ethernet, says John D'Ambrosia, chair of the 802.3ba task force in the IEEE and a senior research scientist at Force10 Networks.

"There are a number of people already who are using link aggregation to try and create pipes of that capacity," he says. "It's not the cleanest way to do things...(but) people already need that capacity."

D'Ambrosia says even though 40/100G Ethernet products haven't arrived yet, he's already thinking ahead to Terabit Ethernet standards and products by 2015. "We are going to see a call for a higher speed much sooner than we saw the call for this generation" of 10/40/100G Ethernet, he says.

According to the 802.3ba task force, bandwidth requirements for computing and core Networking applications are growing at different rates, which necessitates the definition of two distinct data rates for the next generation of Ethernet. Servers, high performance computing clusters, blade servers, storage-area networks and network-attached storage all currently make use of 1G and 10G Ethernet, with 10G growing significantly in 2007 and 2008.

I/O bandwidth projections for server and computing applications, including server traffic aggregation, indicate that there will be a significant market potential for a 40G Ethernet interface, according to the task force. Ethernet at 40G will provide approximately the same cost balance between the LAN and the attached stations as 10G Ethernet, the task force believes.

Core Networking applications have demonstrated the need for bandwidth beyond existing capabilities and beyond the projected bandwidth requirements for computing applications. Switching, routing, and aggregation in data centers, internet exchanges
and service provider peering points, and high bandwidth applications such as video on demand and high performance computing, need a 100 Gigabit Ethernet interface, according to the task force.

"Initial applications (of 40/100G Ethernet) are already showing up, in stacking and highly aggregated LAN links, but the port counts are low," says George Zimmerman, CTO of SolarFlare, a maker of Ethernet physical layer devices.

Zimmerman says 10G is just now taking off in the access layer of large networks and will eventually move to the client side, creating the need for 40/100G in the distribution layer and the network core.

He says the application of 100 Gigabit Ethernet in the core is imminent, and is about two years away in the distribution layer. "Both will be driven by and drive 10G adoption in the access and client end of the network, where today the numbers are still much smaller than the potential," he says

Spec designed for seamless upgrades

The 802.3ba specification will conform to the full-duplex operating mode of the IEEE 802.3 Media Access Control (MAC) layer, according to the task force. As was the case in previous 802.3 amendments, new physical layers specific to either 40Gbps or 100Gbps operation will be defined.

By employing the existing 802.3 MAC protocol, 802.3ba is intended to maintain full compatibility with the installed base of Ethernet nodes, the task force says. The spec is also expected to use "proven and familiar media," including optical fiber, backplanes and copper cabling, and preserve existing network architecture, management and software, in an effort to keep design, installation and maintenance costs at a minimum.

Even though the 802.3ba standard is not expected to be ratified until June 2010, initial interoperability testing will commence later this year, says Brad Booth, chair of the Ethernet Alliance. Public demonstrations will emerge in 2010, and certification testing will start once the standard is ratified, Booth says

The specification and formation of the 40/100G task force did not come without some controversy, however. Participants in the Higher Speed Study Group (HSSG) within the IEEE were divided on whether to include 40G Ethernet as part of their charter or stay the course with 100 Gigabit Ethernet.

After about a month though, the HSSG agreed to work on a single standard that encompassed both 40G and 100G.

"In a sense, we were a little bit late with this," D'Ambrosia says. "By our own projections, the need for 100G was in the 2010 timeframe. We should have been done with the 100G (spec) probably in the 2007-08 timeframe, at the latest. We actually started it late, which is going to make the push for terabit seem early by comparison. But when we look at the data forecasts that we're seeing, it looks to be on cue."

Driving demand for 40/100G Ethernet are the same drivers currently stoking 10G: data center virtualization and storage, and high-definition videoconferencing and medical imaging. Some vendors are building 40/100G Ethernet capabilities into their products now.

Vendors prepare for 100 Gigabit Ethernet

Cisco's Nexus 7000 data center switch, which debuted early last year, is designed for future delivery of 40/100G Ethernet.

"We have a little more headroom, which isn't bad to have when you look at future Ethernet speed transitions coming in the market," says Doug Gourlay, senior director of data center marketing and product management at Cisco. "We're pretty early advocates of the 100G effort in the IEEE.

"[But] the earliest you'll see products from any company that are credible deliveries and reasonably priced: second half of 2010 onward for 40/100G," he adds.

Verizon Business last fall began offering 10G Ethernet LAN and Ethernet Virtual Private Line services to customers in 100 U.S. metro markets. Verizon Business also offers "10G-capable" Ethernet Private Line services.

The carrier has 40G Ethernet services on its five-year road map but no specific deployment dates, says Jeff Schwartz, Group Manager, Global Ethernet Product Marketing. Instead, Verizon Business has more 10G Ethernet access services on tap for later this year.

"We want to get to 100G," Schwartz says. "40G may be an intermediary step."

Once Verizon Business moves its backbone architecture towards 40/100G, products and services will be following, he says.

Spirent Communications, a maker of Ethernet testing gear, plans to release a 40G Ethernet testing module in the second half of this year, and 100 Gigabit Ethernet modules in early 2010, says Tim Jefferson, general manager of the converged core solutions group at Spirent. Jefferson says one of the caveats that users should be aware of as they migrate from 10G to 40/100G Ethernet is the need to ensure precise clocking synchronization between systems -- especially between equipment from different vendors.

Imprecise clocking between systems at 40/100G -- even at 10G -- can increase latency and packet loss, Jefferson says.

"This latency issue is a bigger issue than most people anticipate," he says. "At 10G, especially at high densities, the specs allow for a little variance for clocks. As you aggregate traffic into 10G ports, just the smallest difference in the clocks between ports can cause high latency and packet loss. At 40G, it's an order of magnitude more important than it is for 10G and Gig.

"This is a critical requirement in data centers today because a lot of the innovations going on with Ethernet and a lot of the demand for all these changes in data centers are meant to address lower latencies," Jefferson adds.

Cabling challenges

Another challenge is readying the cabling infrastructure for 40/100G, experts say. Ensuring the appropriate grade and length of fiber is essential to smooth, seamless operation, they say.

"The big consideration is, what's a customer's cabling installation going to look like and what they're looking for to be able to handle that," Booth says. "They are probably going to need to have a parallel fiber capability."

"The recommendations we're making to customers on their physical plant today are designed to take them from 1G to 10G; 10G to a unified fabric; and then address future 40G," Cisco's Gourlay says.

The proposed physical interfaces (PHY) for 40G Ethernet include a range to cover distances inside the data center up to 100 meters, to accommodate a range of server form factors, including blade, rack and pedestal, according to the Ethernet Alliance. The 100 Gigabit Ethernet rate will include distances and media appropriate for data center, as well as service provider interconnection for intra-office and inter-office applications, according to the organization.

The proposed PHYs for 40G Ethernet are 1 meter backplane, 10 meter copper and 100 meter multimode fiber; and 10 meter copper, 100 meter multimode, and 10 kilometer and 40 kilometer single-mode fiber for 100 Gigabit Ethernet.