How To Synchronize FPGAs

[Leroy Tanner]

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

 

[Fred Bloggs]

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

The data stream can be shifted into a serial-in/parallel-out pipelined
register that dumps nibbles into the FPGA array at 1/4 the data rate.

 

[Leroy Tanner]

Hello Fred,

thanks for your reply. Maybe that really is the right way to go. But
nevertheless the problem with synchronization remains somehow. As I am
talking about several GHz's, like around 30 GHz, a 1/4 of the data rate
would still be ca. 8 GHz. That means all four FPGAs should handle that 8 GHz
data synchronously. How can that be achieved? Thanks in advance for your
support.

Regards, Leroy Tanner

 

[Allan Herriman]

Hello Fred,

thanks for your reply. Maybe that really is the right way to go. But
nevertheless the problem with synchronization remains somehow. As I am
talking about several GHz's, like around 30 GHz, a 1/4 of the data rate
would still be ca. 8 GHz. That means all four FPGAs should handle that 8 GHz
data synchronously. How can that be achieved? Thanks in advance for your
support.

A recent design of mine had about 40Gb/s of I/O on a single FPGA.
That was achieved with a number of relatively wide (16 bit) LVDS
buses, running relatively slowly (< 1Gbps per pair).


You haven't said *why* you want multiple FPGAs handling the same data
stream, and why they need to be synchronised so tightly. It would
help if you revealed the nature of the bit stream and the processing
it requires.


BTW, you are using Virtex2-Pro. They have multiple built-in SERDES
devices that work up to 3.125Gbp/s (or 10Gb/s in -Pro-X). It's easier
to route a small number of high speed 100ohm diff pairs around your
board than a large number of lower speed diff pairs.

Regards,
Allan

 

[Iwo Mergler]

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Regards, Leroy Tanner

Hi Leroy,

you'll need to provide a little more information.
Where is the data comming from? In what format?
GHz is a measure of frequency, not data rate.

Current FPGAs run at up to about 500MHz system clock
rates. They tend to have LVDS I/O which reach beyond
10GBit/s per channel in the newest devices.

If you mean LVDS data, you could use one FPGA to split
the stream and then distribute it to the other FPGAs.

Kind regards,

Iwo

 

[Leroy Tanner]

Hello,

The reason why this cannot be realized on a single FPGA is the fact that we
disscussed just the input channels, so far. There is a great amount of
output, too, so I cannot use all of the FPGA's highspeed I/Os (i.e.
RocketIOs) just for the incoming data. In reference to a maximum amount of
20 RocketIOs on the largest Virtex-II Pro I assume that I will only use 4
RocketIOs per FPGA for the purpose of handling the complete 30 Gbit/s. So
that means I must run approximately four FPGAs in parallel. Do you see any
other alternatives?

 

[Allan Herriman]

Hello,

The reason why this cannot be realized on a single FPGA is the fact that we
disscussed just the input channels, so far. There is a great amount of
output, too, so I cannot use all of the FPGA's highspeed I/Os (i.e.
RocketIOs) just for the incoming data. In reference to a maximum amount of
20 RocketIOs on the largest Virtex-II Pro I assume that I will only use 4
RocketIOs per FPGA for the purpose of handling the complete 30 Gbit/s. So
that means I must run approximately four FPGAs in parallel. Do you see any
other alternatives?

This would be so much easier if you told us what you really want to
do.



When I was at Agilent, we had a design that generated and processed
OC768 signals. Each (Virtex-2 6000) FPGA had 40Gb/s input and 40Gb/s
output. Most of that was done using internal 256 bit buses at either
155.52MHz or 200MHz (depending on where it was in the pipeline). I
think it used HSTL or maybe SSTL levels. There were some narrower,
faster (622Mb/s?) LVDS buses between some parts.
That's 80Gb/s total, per FPGA, without using high speed SERDES
interfaces, on parts that are two generations old.

Regards,
Allan

 

[Leroy Tanner]

This would be so much easier if you told us what you really want to
do.

Thanks for the support so far. I'll try to be more precisely and focus on
the decisive factors:

I have got 4 FPGAs (Virtex 2 Pro) available on a printed circuit board. The
Virtex 2 Pro devices offer up to 20 full-duplex 3.125Gbit/s transceivers
(Multi Gigabit Transceivers = MGTs). Because I have a total of - let's say -
20 MGTs and my application arranges for about 16 MGTs for the output data
(let's assume that is a fixed condition), there are just 4 MGTs left for me
to use for the incoming data. So I have a total of 10 Gbit/s (4*2,5 Gbit/s)
on each FPGA and therefore must handle the incoming 30 Gbit/s by parallel
operation of at least 3 FPGAs, better say 4 FPGAs. By the way the incoming
data is received through an Infiniband 12x-Interface, i.e.12 x 1-channel
Infiniband @ 2,5 Gbit/s via channelbonding (=30 Gbit/s). I hope that might
help you in understanding the problem.

Regards, Leroy

 

[Allan Herriman]

Thanks for the support so far. I'll try to be more precisely and focus on
the decisive factors:

I have got 4 FPGAs (Virtex 2 Pro) available on a printed circuit board.

Do you mean you already have the board and now you're trying to work
out how to interface to it?

The
Virtex 2 Pro devices offer up to 20 full-duplex 3.125Gbit/s transceivers
(Multi Gigabit Transceivers = MGTs). Because I have a total of - let's say -
20 MGTs and my application arranges for about 16 MGTs for the output data
(let's assume that is a fixed condition),

there are just 4 MGTs left for me to use for the incoming data.

Plus a few thousand pins of lower speed I/O. Don't forget those, as
their aggregate bw greatly exceeds that of the four MGTs.

So I have a total of 10 Gbit/s (4*2,5 Gbit/s)
on each FPGA and therefore must handle the incoming 30 Gbit/s by parallel
operation of at least 3 FPGAs, better say 4 FPGAs. By the way the incoming
data is received through an Infiniband 12x-Interface, i.e.12 x 1-channel
Infiniband @ 2,5 Gbit/s via channelbonding (=30 Gbit/s).

I hope that might help you in understanding the problem.

Back to 20 questions...

You're still being vague. For instance, does each of the FPGAs need
to see the entire input? Could you arrange things so that three of
the 2.5Gb/s Inifiband channels go to each of the four FPGAs?

Regards,
Allan

 

[Mac]

Thanks for the support so far. I'll try to be more precisely and focus on
the decisive factors:

I have got 4 FPGAs (Virtex 2 Pro) available on a printed circuit board. The
Virtex 2 Pro devices offer up to 20 full-duplex 3.125Gbit/s transceivers
(Multi Gigabit Transceivers = MGTs). Because I have a total of - let's say -
20 MGTs and my application arranges for about 16 MGTs for the output data
(let's assume that is a fixed condition), there are just 4 MGTs left for me
to use for the incoming data. So I have a total of 10 Gbit/s (4*2,5 Gbit/s)
on each FPGA and therefore must handle the incoming 30 Gbit/s by parallel
operation of at least 3 FPGAs, better say 4 FPGAs. By the way the incoming
data is received through an Infiniband 12x-Interface, i.e.12 x 1-channel
Infiniband @ 2,5 Gbit/s via channelbonding (=30 Gbit/s). I hope that might
help you in understanding the problem.

Regards, Leroy

The most important thing, which you totally omitted until this post, is
what does the data bus actually look like?

You mention infiniband (which I'm not too familiar with), but you don't
say whether you have an infiniband transceiver (receiver?) or whether you
actually have to design some kind of infiniband protocol engine into the
FPGA.

You really need to describe your problem in much more detail. If you are
interfacing to some kind of infiniband chip, you should post the part
number so people who want to help you can get some idea of what the chip
spits out.

If you need to accommodate raw infiniband, then some infiniband expert
will have to answer.

In principal, you can synchronize two FPGA's simply by providing them with
synchronized clocks. Doing this right at high speed may be a challenge.

--Mac

 

[Leroy Tanner]

Do you mean you already have the board and now you're trying to work
out how to interface to it?

As a matter of fact I'm "planing" the board but the 4 FPGAs are already
ordered.

Back to 20 questions...

You're still being vague. For instance, does each of the FPGAs need
to see the entire input? Could you arrange things so that three of
the 2.5Gb/s Inifiband channels go to each of the four FPGAs?

Okay, I understand: No, it's not necessary that every FPGA sees the whole
input. The only important thing is that the data remains strictly
interlocked in the time domain. I'm not sure if it is possible to go from a
12x Infiniband Interface (which is a fixed guideline) to three 4x
Interfaces. Infiniband only allows for a 1x, 4x and 12x channel bonding. But
for the demerging of the Infiniband stream one has to use an additional
FPGA...!?

Regards, Leroy

 

[Leroy Tanner]

You mention infiniband (which I'm not too familiar with), but you don't
say whether you have an infiniband transceiver (receiver?) or whether you
actually have to design some kind of infiniband protocol engine into the
FPGA.

The "decoding" of the Infiniband protocol must be done inside the FPGA
fabric. Although I use the Virtex-II Pro FPGA series from Xilinx which
promise to be "fully compliant to the Infiniband protocol" this is more or
less a deception package in my eyes. All protocoll layers must be
implemented in the FPGA, the only thing which is really supplied by the MGTs
is the phyiscal connection with regards to signal level, etc.

If you need to accommodate raw infiniband, then some infiniband expert
will have to answer.

A great idea! Anyone feeling adressed?

In principal, you can synchronize two FPGA's simply by providing them with
synchronized clocks. Doing this right at high speed may be a challenge.

Let's assume we take one FPGA just for the high speed stuff and let the
other FPGAs process the data @ 125 MHz in parallel. Anyway, how is the
synchronisation / deskewing across the FPGA boundaries done? Clock
distribution or transfer protocol?
Regards, Leroy

 

[Leroy Tanner]

you'll need to provide a little more information.
Where is the data comming from? In what format?
GHz is a measure of frequency, not data rate.

The data is coming from a 12x Infiniband interface, which in fact is 12x
2,5Gbit/s handled with a channel-bonding technique. The base technology for
Infiniband is 2.5 Gbit/s differential pairs (LVDS).

Current FPGAs run at up to about 500MHz system clock
rates. They tend to have LVDS I/O which reach beyond
10GBit/s per channel in the newest devices.

You probably refer to the new Xilinx V2PX. They do have 10Gbit/s
transceivers, but unfortunately in the biggest version they also only have
20 of them. I want to interface to Infiniband which is limited to 2,5
Gbit/s per channel (although you get higher rates by channel-bonding them)
but in this respect the 10Gbit/s I/Os are in fact of no use.

If you mean LVDS data, you could use one FPGA to split
the stream and then distribute it to the other FPGAs.

Let's assume we stand an FPGA just for the high speed stuff and let the
other FPGAs process the data @ 125 MHz in parallel. How is the
synchronisation / deskewing across the FPGA boundaries done, anyway? Clock
distribution or transfer protocol?

Regards, Leroy

 

[Nico Coesel]

Hello newsreaders,

For a while I have been confronted with the following task which I find
quite challenging but unfortuantely didn't manage to solve it, yet.
What I want to do is to use 2-4 FPGAs (Xilinx Virtex 2 Pro) together on one
printed circuit board (PCB). They are used to process a large amount of
incoming serial data (data rates of several GHz's). My idea is to handle
that data parallel by the 2-4 FPGAs. But now there arises the problem how to
adequately split the data and how to synchronize the FPGAs among one
another, in particular?
Is it possible or first of all a realistic idea to synchronize multiple
FPGAs in the GHz range? How can this be done without much protocoll
overhead? I would like to do it without applying an extra transfer protocoll
among the FPGAs just for that purpose! Up to this date I didn't find a
proper solution, yet.
Maybe someone can give me a hint? Any ideas how to solve that problem?

Xilinx has some appnotes on this. If memory serves me well, there are
CLKDLLs in the Xilinx FPGA. These make sure the internal clock is in
sync with an external clock. If you apply the same external clock to
all FPGAs (with the same phase), you should be safe. Look for some
CLKDLL appnotes on the Xilinx website.