How to Avoid Idle Signal on Ethernet Cable IEEE 802.3

[Klaus Kragelund]

Hi

I have a system comprised of a computer with a TCP/IP application and a gateway connected to this computer. I need to measure the delay of the gateway, by measuring the delay on the output of the gateway and the signal on the ethernet line

But, the ethernet line is active all the time and I do not have a TCP/IP decoder and it is impossible to trigger on the burried TCP/IP packet.

From what I can learn about the ethernet line, the 100BASE-TX fills the gaps with idle signal. The 100BASE-T4 standard apparently only sends data on the ethernet line when information is actually moved.

Is there any way to "tell" the PC to use a specific standard so its possible to have the scope trigger on a TCP/IP packet?

Regards

Klaus

 

[Tauno Voipio]

Hi

I have a system comprised of a computer with a TCP/IP application and a gateway connected to this computer. I need to measure the delay of the gateway, by measuring the delay on the output of the gateway and the signal on the ethernet line

But, the ethernet line is active all the time and I do not have a TCP/IP decoder and it is impossible to trigger on the burried TCP/IP packet.

From what I can learn about the ethernet line, the 100BASE-TX fills the gaps with idle signal. The 100BASE-T4 standard apparently only sends data on the ethernet line when information is actually moved.

Is there any way to "tell" the PC to use a specific standard so its possible to have the scope trigger on a TCP/IP packet?

Regards

Klaus

It depends, usually not.

This is a hardware function of the interface chip / card.

What you have on the cable is an IP packet in an Ethernet frame. TCP
transport is split in segments encapsulated in some of the IP packets.

If you have to time the TCP transport, my guess is that you do need
protocol analyzers, but even then the scope trigger does not usually exist.

 

[Klaus Kragelund]

It depends, usually not.



This is a hardware function of the interface chip / card.



What you have on the cable is an IP packet in an Ethernet frame. TCP

transport is split in segments encapsulated in some of the IP packets.



If you have to time the TCP transport, my guess is that you do need

protocol analyzers, but even then the scope trigger does not usually exist.

Ok, so an option would be to find an old PC, with older HW, so the idle signalling is not implemented

Thanks

Klaus

 

[Don Y]

Hi Klaus,

I have a system comprised of a computer with a TCP/IP application and
a gateway connected to this computer. I need to measure the delay of
the gateway, by measuring the delay on the output of the gateway and
the signal on the ethernet line

But, the ethernet line is active all the time and I do not have a TCP/IP
decoder and it is impossible to trigger on the burried TCP/IP packet.

What you are looking for is a protocol analyzer. Some *logic* analysers
can be coerced into performing this function.

You can also build a little circuit (PHY+MAC) and carefully control the
traffic on the wire so you *know* that each received packet detected
by your little circuit corresponds to a packet on the other side of
the gateway.

E.g., imagine two of these -- one on each side. You arrange for
<something> to push packets at your gateway. You trigger the
'scope when the first circuit sees the packet (ANY packet). Then,
monitor the output of the second circuit to see when the gateway
has propagated the packet.

As long as the time between packets is much longer than the transit
time across the gateway, you should be able to know that *this*
trigger event is associated with *that* propagated packet (and not
the one before, etc.)

[Think about how you would do something similar with an EIA232
port. Or, worse, a synchronous serial port. Use the "Rx interrupt"
line to signal "character received" -- you don't care *which*
character because you are controlling the content of the link
as well and are arranging for just "measurement" characters to
be passed. Do teh same on the other side of the gateway/bridge.
Compare these two digital signals.]

A simpler way is to write a piece of code to monitor for a particular
packet -- doing nothing else and paying attention to how the code
operates so it takes relatively constant time. Then, have the
code toggle a digital output when it sees a suitable packet.

This is just a software version of the above hardware. Possibly
easier to implement (with a scrap PC or two). You could even do
it in the same PC and have the PC "time" the difference. Of
course, it depends on how much resolution you really need in
your measurements...

 

[Klaus Kragelund]

Hi Klaus,











What you are looking for is a protocol analyzer. Some *logic* analysers

can be coerced into performing this function.



You can also build a little circuit (PHY+MAC) and carefully control the

traffic on the wire so you *know* that each received packet detected

by your little circuit corresponds to a packet on the other side of

the gateway.

Nice idea. I can even take apart the gateway and solder a wire to the Phy on that board, so I do not need external equipment

[snip]

A simpler way is to write a piece of code to monitor for a particular

packet -- doing nothing else and paying attention to how the code

operates so it takes relatively constant time. Then, have the

code toggle a digital output when it sees a suitable packet.

Ok, so I could even have a PC listening in on the line, with a prioritized interrupt, to trigger say the serial port RTS when a package is detected

I just need to turn off all that crap than windows generate in the background. We installed a SW tool to monitor the traffic generated by a computer running windows 7, all kinds of funny stuff going on (probably some from NSA also)

Cheers

Klaus

 

[Nico Coesel]

Ok, so I could even have a PC listening in on the line, with a prioritized interrupt, to trigger say the serial port RTS when a package is detected

I just need to turn off all that crap than windows generate in the background. We installed a SW tool to monitor the traffic generated by a computer running windows 7, all kinds of funny stuff going on (probably some from NSA also)

Linux is the magic word here. I bet there are lots of tools for Linux
which can do the measurement for you. No need to use a scope.

 

[Spehro Pefhany]

Hi

I have a system comprised of a computer with a TCP/IP application and a gateway connected to this computer. I need to measure the delay of the gateway, by measuring the delay on the output of the gateway and the signal on the ethernet line

But, the ethernet line is active all the time and I do not have a TCP/IP decoder and it is impossible to trigger on the burried TCP/IP packet.

From what I can learn about the ethernet line, the 100BASE-TX fills the gaps with idle signal. The 100BASE-T4 standard apparently only sends data on the ethernet line when information is actually moved.

Is there any way to "tell" the PC to use a specific standard so its possible to have the scope trigger on a TCP/IP packet?

Regards

Klaus

Have you tried Wireshark (free)?


Best regards,
Spehro Pefhany

 

[Don Y]

Hi Klaus,

Nice idea. I can even take apart the gateway and solder a wire to the
Phy on that board, so I do not need external equipment

D'oh! That's an even better idea. Or, ask the guys who wrote the code
to push a byte to some (unused) I/O port or memory address that you
can configure a logic analyzer, 'scope, etc. to monitor. (I am
assuming this is the same gateway that you mentioned in a previous
post?)

Ok, so I could even have a PC listening in on the line, with a
prioritized interrupt, to trigger say the serial port RTS when a
package is detected

Only if that PC is running some software that you have written to
run on "baremetal". If you put windows (or damn near any other
COTS OS) in the picture, then you will get lousy data. There
can be large variations in interrupt response times in these
systems.

 

[Don Y]

Hi Jeff,

The "gateway" probably has two ethernet ports.

Actually, it's unclear if that is the case. In previous posts,
Klaus has referenced an ethernet<->EIA485 gateway. As such, there
may not be a device on the far side that responds to ICMP echo
requests!

Or, the gateway may simply NOT forward them! (what is the
EIA485 equivalent of an ICMP echo request/reply??? Is
the protocol massaged *in* the gateway?)

You can simply ping
the computer THROUGH the gateway, to get an approximation of the
delay. The ping time (latency) will be twice the delay through the
gateway, plus whatever the computer adds. Just ping the computer
directly, then ping it through the "gateway" and subtract for the
"gateway" delay. You should get similar results with both ICMP and
UDP ping.

If your "gateway" happens to be a router, you may need to do some
configuring to allow pinging devices on the LAN side from the WAN
side. However, pings should work normally to the WAN side, from the
LAN side without doing anything special.

For better resolution pings try Fping:
<http://fping.sourceforge.net>
<http://www.kwakkelflap.com/fping.html>

Be careful about how much you trust these numbers!

First, they might not have the "resolution" that you need.

Second, you have no way of knowing where the ICMP echo server
is implemented in the network stack/kernel/etc. I.e., no way
to quantify how much of the reported RTT occurs *in* the
remote host, etc.

Klaus hasn't indicated what sorts of numbers he is looking for
or how he wants to use the data he gathers. He could end up
"seeing" propagation delays longer or shorter than what you might
see in a ping.

Ping was never intended to be a "fine grained" tool. So, network
stacks may only give it "token" importance.

[And, while it is "required", you may encounter systems that
disable ICMP services completely! (Think: stealth) ]

OTOH, I'll admit it would be a "cheap" test to perform *today*.
And, depending on the results obtained, may render further
testing unnecessary.

 

[Klaus Kragelund]

Hi Jeff,







Actually, it's unclear if that is the case. In previous posts,

Klaus has referenced an ethernet<->EIA485 gateway. As such, there

may not be a device on the far side that responds to ICMP echo

requests!

The current product we have is a Moxa gateway, with 2 ethernet ports and 2 RS485 ports. The new project is to avoid this cost and make it ourself

Or, the gateway may simply NOT forward them! (what is the

EIA485 equivalent of an ICMP echo request/reply??? Is

the protocol massaged *in* the gateway?)















Be careful about how much you trust these numbers!



First, they might not have the "resolution" that you need.



Second, you have no way of knowing where the ICMP echo server

is implemented in the network stack/kernel/etc. I.e., no way

to quantify how much of the reported RTT occurs *in* the

remote host, etc.



Klaus hasn't indicated what sorts of numbers he is looking for

or how he wants to use the data he gathers. He could end up

"seeing" propagation delays longer or shorter than what you might

see in a ping.

We are just interested in how much delay it is, in the sub ms range resolution

Cheers

Klaus

 

[Don Y]

Hi Joe,

I have not had my morning quart of coffee yet,

*Quart*??! Yikes! I drink a pint of tea "fresh from bed" and
thought *that* was a lot! (OTOH, I then *continue* to drink
another 6-10 pints over the remaining course of the day so I
guess I shouldn't make any claims as to the color of the kettle!)

so I'm not really following
what you want to measure. Is the setup:
Ethernet in -> gateway -> Ethernet out
and you want to measure latency Eth packet to Eth packet?
or
Ethernet in -> gateway -> some other signal out
and measure Eth packet in to assertion of some other signal out?

I believe the latter -- from previous posts. The gateway makes
an "industrial control" (?) EIA485 network available to the
TCP/IP world (or, vice versa)

If it's Eth to signal out, you need some code/interface to monitor
signal out on the Linux box. Use a high enough resolution clock source
in the code and match up the tcpdump packet time stamps with the signal
capture time stamp.

I think you can do what you want with a single Linux box and use that box
as your measure time source.

I think the problem is getting at the signal on the "non-ethernet"
side of the gateway. In a manner that has some small and predictable
latency.

E.g., assuming industrial control, you'd want to know the propagation
delay across the gateway if any aspect of the control algorithm
resided on the "far side". You'd also want to know how consistent
this delay is -- or, modify your protocols to include timestamps
(wrt the industrial control system's idea of time) in messages.

 

[Klaus Kragelund]

Klaus,



I have not had my morning quart of coffee yet, so I'm not really following

what you want to measure. Is the setup:

Ethernet in -> gateway -> Ethernet out

and you want to measure latency Eth packet to Eth packet?

or

Ethernet in -> gateway -> some other signal out

and measure Eth packet in to assertion of some other signal out?

Sorry if that is not clear, it is:

Ethernet in -> Gateway -> RS485 out (Modbus)

So I am interested in the delay of the gateway, so we can estimate the round-robin cycle time for a number of units connected to the gateway, with thedelay of the gateway included in the equation

[snip]

If it's Eth to signal out, you need some code/interface to monitor

signal out on the Linux box. Use a high enough resolution clock source

in the code and match up the tcpdump packet time stamps with the signal

capture time stamp.

The problem is how to sync those two clocks? I thought about making a smallprogram to send the TCP/IP packet from the PC and toogling some other signal at the PC at the same time, so I could measure the time from that toogleto the gateway RS485 port reacted, but that would only be the delay in onedirection

I could assume (assumptions is the.....) that the ethernet TCP/IP latency would be small, and just record the time from sending a package to getting an answer back, but it will probably be wrong by at least 1-10ms (the ping time)

Cheers

Klaus

 

[Klaus Kragelund]

Do you have a Moxa model number?






How sub-msec range do you need? Using Fping, 0.1 msec is possible.

I would be satisfied with 0.1ms resolution, that would be more than good (the unit without a gateway has 25ms turn-around time

Cheers

Klaus

 

[Don Y]

Hi Klaus,

[Argh, what is it with all these blank lines in your posts???]

The problem is how to sync those two clocks? I thought about making a
small program to send the TCP/IP packet from the PC and toogling some
other signal at the PC at the same time, so I could measure the time
from that toogle to the gateway RS485 port reacted, but that would only
be the delay in one direction

Is there any bit of code on the modbus side that is automatically
invoked when "something special" is sent to it? Something that
you can observe independently?

Similarly, some event that you can manually generate on the
modbus side that will cause some "specific" packet/message to
be IMMEDIATELY sent up the wire. I realize you would have to
arbitrate for ownership of that bus (you would have to do
this any time ANY traffic, regardless of direction, was gated
onto the bus) but you could possibly arrange for other traffic
to be quiescent?

Is the content of those messages "time critical" in that they
are used *in* the control loops? Or, just SCADA that you
would *like* to have available, "promptly" (for some idea of
"promptly"). I.e., does the performance of your overall
system degrade as the propagation delay across the gateway
increases?

I could assume (assumptions is the.....) that the ethernet TCP/IP
latency would be small,

Depends on how efficient the gateway is at moving the packets
it encounters. E.g., does it have to do any filtering or does
it just pass everything destined for a particular address/subnet?

and just record the time from sending a package to getting an answer
back, but it will probably be wrong by at least 1-10ms (the ping time)

What does the current product *claim* this delay to be? Or, is
it not specified?

 

[Don Y]

Hi Jeff,

Install an RS-485 loop back adapter in the RS-485 port.
Spew some data on the ethernet port to the RS-485 port.
You should see the data coming back on the ethernet port.
Measure the delay with a protocol analyzer or custom software.

I doubt that will work. Modbus is an application layer protocol.
You're counting on the gateway to see this EIA485 message *destined*
for a particular Modbus "Unit ID" (i.e., address on the 485 wire)
and decide that it should, instead, be routed *out* to the ethernet
side of the fence. To do so, the address field of the message
encountered by the gateway would have to reflect the gateway's
modbus address/unit ID. This would require rewriting the message
"in" the "loopback connector".

Just like looping an ethernet packet back through a router/switch
won't guarantee that it gets *through* the router/switch!

 

[Tauno Voipio]

On 20.9.13 9:00 , Klaus Kragelund wrote:

(Plenty of empty lines and some dicussion clipped)

Sorry if that is not clear, it is:

Ethernet in -> Gateway -> RS485 out (Modbus)

So I am interested in the delay of the gateway, so we can estimate the round-robin cycle time for a number of units connected to the gateway, with the delay of the gateway included in the equation

[snip]

Klaus

IIRC, there is a ping command in Modbus protocol. Measure the signals at
the sending Ethernet machine with Wireshark or tcpdump and scope
the same on RS-485. You'll get the answer by a simple subtraction.
If you want the total delay, the Ethernet trace is all you need.

 

[[email protected]]

Sorry if that is not clear, it is:

Ethernet in -> Gateway -> RS485 out (Modbus)

So I am interested in the delay of the gateway, so we can estimate the round-robin cycle time for a number of units connected to the gateway, with the delay of the gateway included in the equation

The real issue for ethernet to serial converters is how to split the
serial stream to Ethernet packets. The naive approach would be to send
one Ethernet package for each serial character, which is of course
totally unacceptable.

In practice, you haver to assemble several characters to a frame and
send a frame when the frame is full or the last character has been
received. Some old converters waited more than 10 ms with no more
serial characters, before sending the last Ethernet frame.

If the converter understands something about the serial frame, such
knows what is the terminating character in the frame and then send the
last Ethernet frame.

Serial Modbus RTU separates frames with an idle period at least 3.5
character times long, so that is the delay before sending the Ethernet
frame. However, an idle period longer than 1.5 character times is
illegal in Modbus, so you should wait at least that period of time,
before sending the Ethernet frame. The upper level CRC check would
then reject this frame, if such gaps exists within a message.

My experience with reasonable serial speeds (115k2 and below), modern
eth/serial converters send that last frame quite rapidly and any
internal processing delays within the gateway or Eth switches are
small, compared to serial character times (100us@115k2) at reasonable
serial speeds.

The situation was quite different 5-10 years ago, but these converters
have improved considerably.

 

[[email protected]]

Install an RS-485 loop back adapter in the RS-485 port.
Spew some data on the ethernet port to the RS-485 port.
You should see the data coming back on the ethernet port.
Measure the delay with a protocol analyzer or custom software.

What is an RS-485 loopback adapter ?

I know what an RS-232 (connect pins 2 and 3) or RS-422 (Tx+ to Rx+ and
Tx- to Rx-) loopback adapter is, but what is a 2 wire RS-485 adapter ?

Regarding loop back messages on RS-232/422, the Modbus function code 8
(Diagnostics) with subfunction 0 could be used, since it has the same
bit pattern in both request and response, so it can be used in
loopback tests.

 

[[email protected]]

Sorry if that is not clear, it is:

Ethernet in -> Gateway -> RS485 out (Modbus)

So I am interested in the delay of the gateway, so we can estimate the round-robin cycle time for a number of units connected to the gateway, with the delay of the gateway included in the equation

What is your serial speed ?

In any half duplex protocols, such as Modbus RTU, the total throughput
will increase by much less than a factor of 2, when doubling the line
speed, due to various fixed delays, thus there is a diminishing return
when increasing line speed.

I recommend splitting your serial Modbus slaves into several
subnetworks, driven by separate Eth/serial converters driven in
parallel by scanning programs. A 100BaseT backbone can easily handle
the traffic from multiple serial networks. The cost of extra Eth/ser
converters can be compensated by reduced RS-485 cabling costs, if the
slaves are scattered around a larger area.

 

[Tauno Voipio]

The real issue for ethernet to serial converters is how to split the
serial stream to Ethernet packets. The naive approach would be to send
one Ethernet package for each serial character, which is of course
totally unacceptable.

In practice, you haver to assemble several characters to a frame and
send a frame when the frame is full or the last character has been
received. Some old converters waited more than 10 ms with no more
serial characters, before sending the last Ethernet frame.

If the converter understands something about the serial frame, such
knows what is the terminating character in the frame and then send the
last Ethernet frame.

Serial Modbus RTU separates frames with an idle period at least 3.5
character times long, so that is the delay before sending the Ethernet
frame. However, an idle period longer than 1.5 character times is
illegal in Modbus, so you should wait at least that period of time,
before sending the Ethernet frame. The upper level CRC check would
then reject this frame, if such gaps exists within a message.

My experience with reasonable serial speeds (115k2 and below), modern
eth/serial converters send that last frame quite rapidly and any
internal processing delays within the gateway or Eth switches are
small, compared to serial character times (100us@115k2) at reasonable
serial speeds.

The situation was quite different 5-10 years ago, but these converters
have improved considerably.

Modbus has a standard means to pack the frames into TCP transport
(though UDP had been more suitable). The converters I know decode
Modbus frames from serial line and forward them in the standard
way in TCP transport, and the same to the other direction, of course.