Approaching Lightspeed

With the rise of data-hungry applications such as tablet PCs and HDTVs and the increasing popularity of cloud computing, more and more bandwidth is required and chip-to-chip data rates are increasing fast. PCI Express currently allows for 8.0Gbit/s per lane, but this is expected to increase to 16Gbit/s when Gen 4.0 is introduced. This has thrown up a serious problem; chip-to-chip communications will soon reach the physical limitations of the copper interconnects between them.

The problem is copper’s frequency-dependent loss, which is a property of the material itself. This means that as data rates get higher, the maximum length of the interconnects gets much shorter as the insertion loss and return loss increase with respect to the data rate. Signals become degraded and require additional circuitry to compensate, drawing additional power.

Replacing copper interconnects with optical fibre technology is widely touted as the solution to the bandwidth problem. Optical fibres exhibit virtually no loss, and unlike copper their power consumption is relatively independent of length. They are also immune to EMI and can support wavelength division multiplexing.

So far, optical technology has been limited to distances over 10m. However, with the bandwidth problem pressing, optical solutions for chip-to-chip communications are going to be required sooner rather than later. The final spec for PCI Express Gen 4.0 is expected to be released in 2014-5, while the International Technology Roadmap for Semiconductors (ITRS) predicts that for high-speed I/O the data rate per lane will reach 100Gbit/s by 2020.

Public optical demo

Altera claimed an industry-first in March when it demonstrated an optical interconnection to and from one of its FPGAs at the OFC show in Los Angeles. The company has put two of Avago’s 12-channel MicroPOD 850nm vertical cavity surface emitting laser (VCSEL) optical transceiver modules in the same package as a Stratix IV GT FPGA (see Figure 1); at the top right of the package is the transceiver optical sub-assembly (TOSA) and at the bottom right is the receiver optical sub-assembly (ROSA). The total height of the 1932-pin FBGA package is around 10mm. The demo board (see Figure 2) has two MTP connectors for easy communication with the FPGA.

##IMAGE_2_C##
Figure 1: Altera has built a prototype FPGA with integrated optical transceiver modules.

The demonstration itself, developed over one year from the initial concept, shows the FPGA sending and receiving 100Gigabit Ethernet traffic via the optical interconnects, passing through a 100 metre length of multimode fibre (in a loopback configuration). The FPGA generates random packets of varying sizes and sends them back to itself over the optical link. 10 channels, each with a bandwidth of 10.3125Gbit/s make up the 100 Gigabits required, so 2 of the channels on each transceiver are unused in this demo.

The bit error rate (BER) is 10-12 or less. According to Craig Davis, Product Marketing Manager for Altera Europe, before OFC the demo was run continuously in the lab for 72 consecutive hours with zero errors occurring. Digital diagnostics monitoring (DDM), such as module case temperature and laser bias current, detects potential issues and prevents link loss.

##IMAGE_3_C##
Figure 2: Altera demonstrated its optical FPGA at OFC using this demo board.

Design challenges

Integrating the optical transceiver modules directly into the FPGA’s package minimises the distance between the I/O pads of the chip and the optics, helping to minimise signal degradation and jitter. The integration also reduces footprint. However, there were considerable challenges to be overcome when integrating the modules. Their reliability in particular caused problems.

“Laser drivers and receivers do not have the same reliability as the FPGA,” said Davis. “They were going to be soldered down on the substrate… we had to make them modular, so we could replace the optical modules without removing the FPGA from the board.” Socketising the optics also means the modules are not limited to any manufacturer or wavelength, for the purposes of future development.

Another challenge was keeping the optical modules within their standard operating temperature range, 0 to 70 degrees C. A custom heat sink and fan arrangement had to be constructed to cool things down sufficiently (see Figure 3).

##IMAGE_4_C##
Figure 3: A heat sink and cooling arrangement constructed in the lab keeps the temperature of the optical modules within safe operating limits.

The power consumption of the optical sub-assemblies used in the demonstration is between 2 and 3W, for both (Avago’s spec for the modules gives their consumption at 125mW per channel, and 10 input and 10 output channels are used). The power required by the optics is traded off against the reduced need for equalisation. Copper-based systems require equalisation to compensate for inter-symbol interference, data-dependent jitter, return loss and crosstalk. A typical system may contain a feed-forward equaliser (FFE), continuous time linear equaliser (CTLE), and/or a decision feedback equaliser (DFE).

“Copper needs adaptive equalisation and DFE, and all these things take power. As you increase the data rate the losses get much greater,” Davis observed. Optical technology does not require the equalisation that copper does because the material doesn’t exhibit the same losses.

It’s difficult to compare the power consumption of copper and optical interconnects directly, because the power used to send data over copper depends of a variety of factors including equalisation schemes, materials and levels. However, with the main driving factor being bandwidth, power consumption may be a moot point anyway. FPGA transceiver data rates can currently achieve 28Gbit/s at the 28nm process node, and according to Davis, these devices would often have up to ten inches of copper track connecting them to an optical module anyway. Cutting out the copper by placing the optics in the package gets rid of the bandwidth bottleneck without significantly impacting on power consumption.

At the time of writing, copper is still seen as more cost effective for link speeds of 10Gbit/s at distances of a metre or less, but as data rates increase, this cannot continue to be the case. Optical interconnects between chips are inevitable and although the Altera demo is an important first step, there are still a lot of factors which are yet to be addressed. Will existing standards such as PCI Express be extended to address chip-to-chip optics, or will new standards bodies emerge? Will one wavelength dominate? What about wavelength multiplexing? Altera uses MTP connectors in its demo, but will new standard connectors be developed that better meet the needs of chip packaging? And what about integrating optical links within PCBs?

Altera is certainly keen to stress that its demonstrator is just that, a demonstrator, and that products benefitting from this technology are still very much in the future. Applications that could be initial adopters of chip-to-chip optical communication include broadcast, datacoms and telecoms, test and measurement, medical electronics and military networks.

“You may see more of this in areas where board design is a big proportion of the cost and the volumes are low, perhaps in the scientific community,” Davis said. He also cited high speed communications systems as a potential first adopter since this application would not be in the top tier volume-wise, but still require very high speed signals.