How to move binary data into a PC?

[Virgil Smith]

Can anybody give me advice on capturing a raw binary data stream into a PC?

I'm working on a project to develop an instrumentation receiver for
monitoring
signal strength from a number of remote transmitters. The front end (tuner,
mixer, digitizer, etc.) is working pretty well, but the next step is giving
me trouble. I will be shifting out data from an A/D converter, and need to
get it into a PC somehow, time tag each sample, store records in a logging
file and every few days read the accumulated stuff out from a host computer
(FTP or Telnet over the Internet, I'm thinking). There will be several of
these data-gathering stations operating unattended, spread out over several
miles. Data files should be no more than a few Mbytes.

I thought this would be straightforward, but I can't come up with a simple
way
to handle the PC interface. Is it really so difficult to do this, or am I
missing something? I hope it's the latter, and somebody here can point me in
the right direction.

I'm more a hardware guy than a programmer, so I'm looking for something
simple.

Thanks for any advice.

-vs-

 

[Leon]

Can anybody give me advice on capturing a raw binary data stream into a PC?

I'm working on a project to develop an instrumentation receiver for
monitoring
signal strength from a number of remote transmitters. The front end (tuner,
mixer, digitizer, etc.) is working pretty well, but the next step is giving
me trouble. I will be shifting out data from an A/D converter, and need to
get it into a PC somehow, time tag each sample, store records in a logging
file and every few days read the accumulated stuff out from a host computer
(FTP or Telnet over the Internet, I'm thinking). There will be several of
these data-gathering stations operating unattended, spread out over several
miles. Data files should be no more than a few Mbytes.

I thought this would be straightforward, but I can't come up with a simple
way
to handle the PC interface. Is it really so difficult to do this, or am I
missing something? I hope it's the latter, and somebody here can point me in
the right direction.

I'm more a hardware guy than a programmer, so I'm looking for something
simple.

A suitable MCU interfaced to the PC via RS-232?

Leon

 

[Tim Wescott]

A suitable MCU interfaced to the PC via RS-232?

Leon

That was my suggestion. Best bet would be to format it in ASCII,
perhaps in a comma-delimited format that could be directly imported into
Excel or compatible programs.

This would be easy to do with just about any micro controller -- a PIC
or an 8051 would be the most likely choices.

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Posting from Google? See http://cfaj.freeshell.org/google/

 

[Virgil Smith]

That was my suggestion. Best bet would be to format it in ASCII,
perhaps in a comma-delimited format that could be directly imported into
Excel or compatible programs.

This would be easy to do with just about any micro controller -- a PIC
or an 8051 would be the most likely choices.

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Posting from Google? See http://cfaj.freeshell.org/google/

Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

-vs-

 

[Luhan]

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

I don't know what else you may expect to be around. RS-232 serial
ASCII is about the most common computer connection that has ever been
around. It has been used, as far as I know, by about every computer
company in history. Although some (laptop) computers today come
without COM ports, you can always get some plug-in card to provide it.

Luhan

 

[Nico Coesel]

Can anybody give me advice on capturing a raw binary data stream into a PC?

I'm working on a project to develop an instrumentation receiver for
monitoring
signal strength from a number of remote transmitters. The front end (tuner,
mixer, digitizer, etc.) is working pretty well, but the next step is giving
me trouble. I will be shifting out data from an A/D converter, and need to
get it into a PC somehow, time tag each sample, store records in a logging
file and every few days read the accumulated stuff out from a host computer
(FTP or Telnet over the Internet, I'm thinking). There will be several of
these data-gathering stations operating unattended, spread out over several
miles. Data files should be no more than a few Mbytes.

I thought this would be straightforward, but I can't come up with a simple
way
to handle the PC interface. Is it really so difficult to do this, or am I
missing something? I hope it's the latter, and somebody here can point me in
the right direction.

I'm more a hardware guy than a programmer, so I'm looking for something
simple.

Depending on your budget, you may consider a digital I/O card which is
connected to the A/D converter directly. These cards are available in
several varieties (USB, PCI, PCMCIA). These cards usually come with
their own API which is (usually) easy to deal with.

I think this is the shortest route. You'll have the data inside the PC
without having to develop another 'in between' device and debugging a
PC application is much easier than software in a microcontroller.

 

[Leon]

Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

It's not difficult to write some software in C or any other suitable
programming language that reads data from the PC serial port and
outputs it to a file, with time/date stamps. I'd do it from DOS with a
program written in gcc.

Leon

 

[Sjouke Burry]

Can anybody give me advice on capturing a raw binary data stream into a PC?

I'm working on a project to develop an instrumentation receiver for
monitoring
signal strength from a number of remote transmitters. The front end (tuner,
mixer, digitizer, etc.) is working pretty well, but the next step is giving
me trouble. I will be shifting out data from an A/D converter, and need to
get it into a PC somehow, time tag each sample, store records in a logging
file and every few days read the accumulated stuff out from a host computer
(FTP or Telnet over the Internet, I'm thinking). There will be several of
these data-gathering stations operating unattended, spread out over several
miles. Data files should be no more than a few Mbytes.

I thought this would be straightforward, but I can't come up with a simple
way
to handle the PC interface. Is it really so difficult to do this, or am I
missing something? I hope it's the latter, and somebody here can point me in
the right direction.

I'm more a hardware guy than a programmer, so I'm looking for something
simple.

Thanks for any advice.

-vs-

Could you link them to a wifi wireless network?
dumping data directly to the remote shared harddisk?
You might have to ask a LINUX guru ,because the
number of open shares in XP might be limited.
The range ,when using directional arials might
be several miles.(according to a thread a few days
ago).

 

[Virgil Smith]

Could you link them to a wifi wireless network?
dumping data directly to the remote shared harddisk?
You might have to ask a LINUX guru ,because the
number of open shares in XP might be limited.
The range ,when using directional arials might
be several miles.(according to a thread a few days
ago).

Interesting idea, but in this case we're talking about several tens of
miles, and in one
case, maybe 100.

-vs-

 

[scada]

a need several am


Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

-vs-

csv (comma separated value) files can be imported into excel, these are just
ascii comma delimited files. A vb written program can pick the data from
your serial port and place it into the excel spreadsheet. If you need graphs
larger than excel's limited 64K, have a look at www.dplot.com for around $30
it's a great deal - i've been using it for all my data acquisitions for
several years now. As far as the fate of serial ports, I have the same
concerns. there are several serial to USB dongles out there that claim to be
"virtual" serial ports on the pc. I plan to look into Microchips PIC, they
have USB & USART's built in. Good luck...

 

[John Jardine.]

[brevity clip]

Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

-vs-

With 100% conviction I'd suggest that PowerBasic. Suspect the needed time
stamped log file program would take a whopping 10 lines or so.
I'd stick with RS232. It has a massive, diverse user base (=money to be
made)hence will receive continued support. In the meanwhile other comms
fluff will no doubt come and go.
john

 

[[email protected]]

csv (comma separated value) files can be imported into excel, these are just

ascii comma delimited files. A vb written program can pick the data from
your serial port and place it into the excel spreadsheet. If you need graphs
larger than excel's limited 64K, have a look at www.dplot.com for around $30
it's a great deal - i've been using it for all my data acquisitions for
several years now. As far as the fate of serial ports, I have the same
concerns. there are several serial to USB dongles out there that claim to be
"virtual" serial ports on the pc. I plan to look into Microchips PIC, they
have USB & USART's built in. Good luck...

I read that some of these rs232-usb dongles don't adhere to standards properly
in way that makes them troublesome.
Otoh, with PIC one can run the show oneself

 

[[email protected]]

Can anybody give me advice on capturing a raw binary data stream into a PC?

I'm working on a project to develop an instrumentation receiver for
monitoring
signal strength from a number of remote transmitters. The front end (tuner,
mixer, digitizer, etc.) is working pretty well, but the next step is giving
me trouble. I will be shifting out data from an A/D converter, and need to
get it into a PC somehow, time tag each sample, store records in a logging
file and every few days read the accumulated stuff out from a host computer
(FTP or Telnet over the Internet, I'm thinking). There will be several of
these data-gathering stations operating unattended, spread out over several
miles. Data files should be no more than a few Mbytes.

Distance: 10-100 miles
Amount: few MBytes/week?

Can you tell what kind of link you will have?, IP connectivity with ethernet
interface. Phone lines?, Packet radio?

The units could get by with some simple mcu.

As for getting the data into a pc. You could utilise 'perl'. Which isn't
pretty but will mangle the data in almost any way you like.

RS232 is also quite feature safe.

What kind of data is collected ..?

 

[Marte Schwarz]

Hi Virgil,

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

buy a USB-cellphon cable (I use the Samsung E700). Cut the Plug and use the
cables directly to power my µC from USB and habe a UART with 3V3 Level. What
do we need more? Yes a 5V to 3V3 LDO and a few caps.

Marte

 

[Virgil Smith]

[brevity clip]

Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

Also (secondary consideration), I was wondering if it's wise to rely on
conventional serial and/or parallel ports to be around in several years.
I would like this project to have a fairly long life, and I can foresee
having
to replace components (like laptop computers) as time goes by. Other, less
knowledgeable folks may be running things eventually.

-vs-

With 100% conviction I'd suggest that PowerBasic. Suspect the needed time
stamped log file program would take a whopping 10 lines or so.
I'd stick with RS232. It has a massive, diverse user base (=money to be
made)hence will receive continued support. In the meanwhile other comms
fluff will no doubt come and go.
john

I've heard a lot of good things about Powerbasic, and I may go that way.
It's inexpensive, seems
well-established and has all the capability I need for this job. Something
one could base a project
on with some confidence!

-vs-

 

[Virgil Smith]

PC?

Distance: 10-100 miles
Amount: few MBytes/week?

Can you tell what kind of link you will have?, IP connectivity with ethernet
interface. Phone lines?, Packet radio?
IP/Ethernet


The units could get by with some simple mcu.

Yes. I was thinking of one of the Microchip PIC family with USB capability.
But if the budget
can take it, I can just use a commercial data acq module for each site.
DataIO, Dataq, Onset
and others all make units that should do.

As for getting the data into a pc. You could utilise 'perl'. Which isn't
pretty but will mangle the data in almost any way you like.

Haven't used Perl, but I've heard about it for years. What's it like? Isn't
it basically a scripting language?

RS232 is also quite feature safe.

What kind of data is collected ..?

Field strength measurements multiplexed over 3-5 channels. Sample rate will
be about 1/minute, with
10-bit precision.

 

[[email protected]]

Distance: 10-100 miles
Amount: few MBytes/week?
Connectivity: IP/ethernet

Yes. I was thinking of one of the Microchip PIC family with USB capability.
But if the budget
can take it, I can just use a commercial data acq module for each site.
DataIO, Dataq, Onset
and others all make units that should do.

Why usb when your datapipe is ip/ethernet in the end? it would only add a cost
of 20 EUR to do the usb->ether.

Tuner -> Mixer -> A/D -> Mcu -> NIC-Ethernet -> Magnetics
|
|

Mcu: AVR, PIC
Nic: RTL8019 (Asfair realtek have some nic that does 8-bit)
And you need "magnetics" aswell.

Another option is a mobilephone + mcu. As most mobiles talk serial in some
way or another. (ex SE CDU-11 datacable is a rs232-usb adapter in reality
asfaik).

Haven't used Perl, but I've heard about it for years. What's it like? Isn't
it basically a scripting language?

It's a mix between shellscript and C languagewise. But it's nice because it
allows you to glue together programs. And let's you concentrate on your
goal. Provided the task is relative small.

Field strength measurements multiplexed over 3-5 channels. Sample rate will
be about 1/minute, with
10-bit precision.

Sound like some weather, possible thunder science

I looked into thunder detectors when I worried it would blow up something

(2x aerial -> amplifier -> soundcard or real 200 kHz adc, then triangulation

 

[Le Chaud Lapin]

Right. I was thinking along those lines myself. The real sticky part
for me is how to manipulate the bytes in the PC environment. What software
packages do people use to organize RS-232 inputs into records, files, etc.
complete with time tags? Power Basic, C and Python have all been suggested
to me, but none with any conviction.

RS-232, by far, is the easiest way to go on a PC, and frankly, you
really don't have any alternatives.

USB is not what people often think - you can't just whip up some code
to talk to a USB port on a PC. The plug and play facility is
misleading - to talk to a raw USB port requires considerable skill in
driver-level development, because, techncially, USB hardware requires
device drivers to present an interface inside the kernel, upon which
other device drivers ride, and finally, the kernel presents a
user-level interface that you can program against using your favorite
language such as C or BASIC. So if the USB device is a digital camera,
the interface that at the user coding level will see something that
expects to behave like a digital camera. If you just want to send
bytes back and forth as you would for RS-232, forget it, you will
suffer, unless you are using a configuration that mimics an RS-232
serial port, but at that point you might as well use a real serial
port. (This is slightly not true but...).

So with RS-232, I would do this:

* Connect modems on each remote module's serial port. Connect modem to
central computers serial port. Since distance is so far, use
conventional external modems with POTS. If distance were shorter, you
could use cold cable with no dial tone and it would still work.

* Pick your favorite compiler. Microsft has done a stellar job in
creating these free compilers:
http://msdn.microsoft.com/vstudio/express/. My favorite is C/C++. You
probably want the one for BASIC.

Here's steps in C. There wiill be direct analogs in BASIC, because all
the BASIC API's in Windows do is simply call the underlying system
API's (which are written in C).

* On central computer, Open serial port- CreateFile()

* Set format for transfer 9600-115,200, 8, N, 1, (don't remember
function).

* Dial out ATDT+1 617 555-1212 pause to allow connnection -
WriteFile().

* Open a data file - CreateFile() again

* Read byte or bytes from port - ReadFile()

* Call system function to get time for each record - GetSystemTime()

* Write collected record with time stamp into data file - WriteFile()

* After finished, close data file - CloseHandle().

* Close serial port - CloseHandlle() again

Store records in format of your choice, like CSV, etc. Note that there
is no "magic" to formats. Pretty pictures won't automatically jump up
on the screen just from choosing a format. Human beings must give
semantics and detailed instructions on what to do with the data before
anything fancy happens.

Use Hyperterminal to test your embedded module before you write your
central-PC application by typing commands directly from Hyperterminal,
through cable, to the module (which means u will need ASCII command
set).

-Le Chaud Lapin-

 

[Virgil Smith]

RS-232, by far, is the easiest way to go on a PC, and frankly, you
really don't have any alternatives.

USB is not what people often think - you can't just whip up some code
to talk to a USB port on a PC. The plug and play facility is
misleading - to talk to a raw USB port requires considerable skill in
driver-level development, because, techncially, USB hardware requires
device drivers to present an interface inside the kernel, upon which
other device drivers ride, and finally, the kernel presents a
user-level interface that you can program against using your favorite
language such as C or BASIC. So if the USB device is a digital camera,
the interface that at the user coding level will see something that
expects to behave like a digital camera. If you just want to send
bytes back and forth as you would for RS-232, forget it, you will
suffer, unless you are using a configuration that mimics an RS-232
serial port, but at that point you might as well use a real serial
port. (This is slightly not true but...).

So with RS-232, I would do this:

* Connect modems on each remote module's serial port. Connect modem to
central computers serial port. Since distance is so far, use
conventional external modems with POTS. If distance were shorter, you
could use cold cable with no dial tone and it would still work.

* Pick your favorite compiler. Microsft has done a stellar job in
creating these free compilers:
http://msdn.microsoft.com/vstudio/express/. My favorite is C/C++. You
probably want the one for BASIC.

Here's steps in C. There wiill be direct analogs in BASIC, because all
the BASIC API's in Windows do is simply call the underlying system
API's (which are written in C).

* On central computer, Open serial port- CreateFile()

* Set format for transfer 9600-115,200, 8, N, 1, (don't remember
function).

* Dial out ATDT+1 617 555-1212 pause to allow connnection -
WriteFile().

* Open a data file - CreateFile() again

* Read byte or bytes from port - ReadFile()

* Call system function to get time for each record - GetSystemTime()

* Write collected record with time stamp into data file - WriteFile()

* After finished, close data file - CloseHandle().

* Close serial port - CloseHandlle() again

Store records in format of your choice, like CSV, etc. Note that there
is no "magic" to formats. Pretty pictures won't automatically jump up
on the screen just from choosing a format. Human beings must give
semantics and detailed instructions on what to do with the data before
anything fancy happens.

Use Hyperterminal to test your embedded module before you write your
central-PC application by typing commands directly from Hyperterminal,
through cable, to the module (which means u will need ASCII command
set).

-Le Chaud Lapin-

This is a very enlightening discussion and I thank you for the pithy project
outline.
I hadn't thought how useful Hyperterminal might be as a debugging tool,
until you
mentioned it. Good one!

My job will be done once I get everything into a standard file format (CSV,
probably).
Somebody else will worry about making the pretty pictures for reports and
presentations.

My regards to everyone who has chimed in here. I appreciate the help.

-vs-
..
I'll

 

[Nico Coesel]

Yes. I was thinking of one of the Microchip PIC family with USB capability.
But if the budget
can take it, I can just use a commercial data acq module for each site.
DataIO, Dataq, Onset
and others all make units that should do.

I recommend to stay away from PIC processors. Been there, will never
go there again. Too cumbersome to program and way too sensitive to
static discharge.

A TI MSP430F149 uC and a CP2102 USB tot serial chip (from Silabs) make
a nice and simple combo.

The adavantage of the MSP430 series is that you don't need a
programmer to program them. They can be programmed directly using the
serial port. Software can be written in C using a GCC compiler (look
for mspgcc).