Connect with us

need data capture, stat analysis, graph display software for PC

Discussion in 'Electronic Design' started by colin, Mar 22, 2007.

Scroll to continue with content
  1. colin

    colin Guest

    Hi,
    I got my PIC producing 10mb data/hr,
    now I need something to receive the data
    via bluetooth store it and do some statistical analysis
    on it and display the results etc.

    Before I reinstall my old copy of MSVC C++ I wondered
    if anyone knows of a quicker way to get results.
    it will probably take me a while to do it with C++ doing all the coms and
    windows and maths and graph stuff with mfc,
    but ive looked at a few suggestions made like scilab but it doesnt seem to
    be all the flexible,
    maybe would be realy good to display the final results in 4d or whatever.

    there are also some realy hi price packages too,
    It might take me longer to wade through all the different ones
    then learn it, then find out its not actually so easy to do exactly
    what I want after all becuase its not flexible enough.

    maybe I need a good C++ library wich maybe even replaces MFC ?

    Does anyone have any usefull experience to share ?

    basically I have an experiment wich measures the signal rotating at 3rps
    and I need to determine direction from this wich will vary cyclicly
    over 24hours.

    The signal is 1000 times lower than the noise however,
    also considerable number of points will be lost due to glitches etc.
    The PIC already does some DSP to recover the signal up to this low level.

    or maybe is there another forum wich might specialise in this stuff ?

    Colin =^.^=
     
  2. John Barrett

    John Barrett Guest

    C# has (in the latest release) good serial suppurt, which shouldnt have a
    problem linking to serial over bluetooth, and there are a number of decent
    and FREE 3D graphics interfaces for C# -- DirectX and OpenGL at the low
    level, Axiom and others at a higher level (though those are more scene
    oriented)... in any case... C# eliminates the majority of the crap that MFC
    or any C++ UI will put you through creating a decent UI

    We can take this off group if you'd like to discuss it more.
     
  3. colin

    colin Guest

    thanks, ive not looked at C# yet maybe its time to,
    is it similar enough to c++ to pick up quickly ?

    Il go look for some online c# references.

    its not that far off topic here is it? it is data from an electronic
    experiment.
    maybe if we get involved in how to combine al the differnt windows
    attributes such as scroll copy/paste in one window ..

    Colin =^.^=
     
  4. Would Python be any good at this sort of thing?



    martin
     
  5. Wim Ton

    Wim Ton Guest

    If you can get some example code that will write the serial results into a
    delimited ASCII file, you can do a lot of processing and graphing in Excel.
    Not elegant, but it saves you from learning new packages or languages

    Wim
     
  6. EdV

    EdV Guest


    Serial Port Import to MS Excel:
    http://www.business-spreadsheets.com/solutions.asp?prod=112

    I haven't tried this one but it sure looks doable.
     
  7. colin

    colin Guest

    Hi, thanks, Ive arranged the data in CSV (comma seperated variables),
    as this seemed most widely used, wich comes in through the com4 port.

    Im using bluesoleil software on the PC for the bluetoth dongle,
    wich works fine with my rs232->bluetooth device except if the
    remote system gets power cycled or the PC reset for any reason
    it requires manual intervention to restablish the link.
    this is quite irritating, all the other classes of bluetooth
    devices seem to have options to start automatically.

    I could save the data in DOS window with 'type com4>file.txt' but thats not
    so elegent.
    would be nice to timestamp the data with accurate time too.
    atm im just loking at it on a dos screen,
    as hyperterminal realy sucks trying to see large chunks of data.

    I would need to do FFT on the data to find the direction,
    then do FFT on that again to find the cyclcic change in direction,
    its probably not that difficult to do,
    in fact I would just do it all in the PIC if it had more than 16k of ram!
    it has a DSP engine after all ! although ive done it for a limited period
    but I realy need to do average over fixed time span rather than just a
    running average so need acess to al the points.

    I also need to do standard deviation at both FFT steps and filter out any
    glitches,
    again this isnt to dificult in theory as such,
    but im not sure a spreadhseet would cope with 10e6 data points,
    I dont realy like the idea of spreadsheets that much tbh,
    so ive never realy used them.
    Although Ive programmed in C++ for many many years.

    Im not sure if the higher level packages would let you reject points of data
    or then let you do an FFT on it.

    A graphical display of the data would be nice although a final list of the
    stats would be ok albeit a bit boring for all the work gone into it, the
    graph would at least give me some visual confirmation of the stats too.

    Ive just installed the C# from microsoft as it seems to be free !
    C# looks promising too.

    Colin =^.^=
     
  8. Wim Ton

    Wim Ton Guest

    Probably not, but 10e4 is easily done
    The nice thing of spreadsheets is, that you can create a graph in half a
    dozen mouse-clicks. You can even do crude DFT's faster then learning an
    advance math package :p
    You can split the processing in several stages:
    Remove glitches
    FFT
    etc.
    Each of which are very manageble in whatever language. If you wite the
    intermediate results in a CSV, you can have a quick peek if the result looks
    halfway decent.
    I would suggest to use a standard library for the FFT, so look if there is
    one for C#

    Wim
     
  9. John Barrett

    John Barrett Guest

    techincally you could use ANY language that can cope with the serial
    interface, and you can extend python with C/C++ to pull in the needed
    graphics capability, but there really isnt a point -- C/C++/C# have the
    graphics libraries available off the shelf, as well as loads of code
    available for massaging the data before display -- I dumped C++ for C# after
    learning Java -- no way I'm going to deal with anything less than a fully
    object oriented interface ... python leaves something to be desired there,
    and I dont like its indentation-sensitive syntax
     
  10. John Barrett

    John Barrett Guest

    C# is a blend of C++ and Java -- you wont have any problems picking it up
    very quickly -- plenty of example code out there --
    http://www.codeproject.com is a good place to start

    If you dont want to get hooked in to the MS world -- use Java -- Java3D can
    handle the data display and the netbeans IDE is pretty comprehensive (you
    can download it at java.sun.com free)
     
  11. John Barrett

    John Barrett Guest

    Ahh you must have found the "express" version or whatever they are calling
    it lately -- that'll get you there :)

    feel free to drop email on me if you have C# specific questions.
     
  12. colin

    colin Guest

    yep it was quite surprising to find it, I expected it to be limited.
    wonder why they giving it for free, must be becuase it ties you into windows
    easily with .net.

    admitedly it wouldnt let me instal it to my e: drive where I had space
    it still decided to try and plonk 1.3gb on my c drive wich is now full,
    even though I specified a dir on e:

    lets you generate an application in 60 seconds
    wich is fine if all you want to do is have a button
    wich changes a label to "hello world" when you press it.

    I already have com port open so it puts lines from com4 into where the hello
    world was
    when you preess the button lol, so thats 30% of the way there.

    I think il just have to massage the data and interpolate between points wich
    I have to reject as glitches.

    There are some FFT libraries too thats another 30%,
    google came up with one that looked like it would do,
    dont know if there are any recomendations available?
    although I dont fully understand the FFT terminology used exactly,
    I know how to use a DSP to do one though lol.
    does the PC have real DSP capabilities wich it uses or just a normal but
    fairly quick FPU?
    im runing with an AMD64bit CPU.

    I have also moved the com port to a new task/thread for reading
    and handled acess denied exceptions when it wont open.
    atm im looking for a good way to pass the data from the com task to the
    processing task.
    I gues a stream or a pipe or whatever its equivalent is in c#.net
    Il probably find that too, just take a bit more time looking than the other
    stuff did so far.

    Im sure il find a c# graph library too,
    again maybe has some prefered ones.

    Colin =^.^=
     
  13. thanks for that, I'm not "a PC programmer" but quite happy writting
    small stuff for 8 bitters, I was just thinking of a simple solution in
    the PC world, that would have helped the OP.


    martin
     
  14. John Barrett

    John Barrett Guest

    FPU based -- no on-board DSP :)

    for the data transfer -- look at the System.Collections.Generic namespace --
    create a class that represents a block of FFT ready data.. have the com port
    thread assemble the block, and when ready.. push the block into a typed
    queue (generics are like C++ templates.. let you create collections of typed
    objects with full compile time type checking)

    use the "lock" statement to wrap your push and pull methods so that access
    to the queue is serialiized, and optionally create an event to notify the UI
    code that there is a block of data ready to process. (or use a timer to poll
    for data -- whatever you are comfy with) Note that C# gets pissy when you
    attempt to access UI elements from a thread other than the one that created
    the UI -- look at delegates and the Invoke method to work around the issue
    if you need the background thread to access the UI.

    You might want the FFT and any other data crunching on the com thread or
    even a seperate cruncher thread so the UI just has to deal with displaying
    ready-to-render data.

    sounds like you are off to a running start, though !! glad to see you are
    slipping into C# so quickly :)

    (ohh and the Mono system for *nix will run dotNet code -- so you really
    arent all that locked in to MS with C# any more )
     
  15. colin

    colin Guest


    The com thread needs to be real quick as it needs to timestamp each line to
    10ms resolution if possible,
    itl just do that and also split it into seperate chanells for fft data and
    housekeeping data, errors etc.
    and save it in files. the PIC is short of a RTC with proper time.
    the fft thread has to read any old data stored in files then add the new
    data from the com port
    thread then do number crunching every so often on the data collected so far.
    thanks that 'lock' sounds like what I need,
    of course I could just use the files to transfer data ..
    just make them each hold chunks of 10 minutes or so they are finished
    with by the com thread before the fft thread gets to them,
    but the lock would make me free to make them longer.
    im sure theres probably simple pipe type class wich uses the thread lock.
    yeah thats just like mfc, it used temp data objects wich arnt garanteed to
    not have been destroyed when other threads are running.
    Cool thanks, yeah I get up to speed real quick I gues,
    but only if i can find the enthusiasm though and then thats either on or off
    with little inbetween
    I just bualk at the idea of trying to go through loads of new things trying
    to evaluate them,
    ive tried that before and it can realy suck.

    Colin =^.^=
     
  16. John Barrett

    John Barrett Guest

    I was thinking more realtime than logging to files -- think about having the
    com thread pump the data to a writer thread in that case :) sounds like you
    may end up with 3 or 4 threads sharing the load, which is a good thing
    IMO -- as threads that are idle dont take resources -- event driven makes
    life much simpler

    might also look at the background worker class -- it encapsulates a lot of
    the crap for starting a worker thread as needed and getting notified when it
    completes
     
  17. colin

    colin Guest

    I dragged that very class right of the IDE toolbar into the project.
    same with the serial class.

    3 or 4 threads is probably about right.

    I need to save the data regardless anyway.
    I could have the analysis done after all the data is gathered but realy
    would like to see how well its progressing so i can catch anything
    going wrong and not waste days colecting duff data.

    I found a Queue wich can transfer objects between threads
    took me a while to figure out I needed to actualy create an instance of it
    with new.
    works but I gues I need to protect it with lock and I need to have something
    to wait on.
    I tried the messaging thing but it complains its not installed on my pc.
    the sockets seem primitive as they stil rely on byte streams,
    although come to think of it that is actually all what I have anyway.

    cant beleive they havnt put in some sort of decent inter thread message
    queue.
    although theres a few examples of how to implemet one about.
    cant even send them as simple windows event messages like with mfc.

    Colin =^.^=
     
  18. John Barrett

    John Barrett Guest

    ya -- they dont make the collection classes thread-safe for performance
    reasons -- they figure anyone that needs it thread safe can create a wrapper
    class derived from the collection and override the key methods to implement
    locking

    If by messaging, you are talking about MSMQ -- thats a bit overkill for
    this -- just create a protected queue class and go from there -- its easy
    enough to add events to the class so that the reciever can get notification
    when data is available. You dont need to "wait" on anything -- just have
    each class set up its event handlers in the constructor, and leave the
    instance to sit until an event occurs.

    "simple windows event messages" -- why get the OS involved with the
    notifications ?? I can understand how you would feel that WM is simple given
    your MFC background, but you are pushing your message out of process when it
    really doesnt need to. using WM like that was a "lazy" solution for people
    that didnt want to write thread safe code, or didnt undestand mutual
    exclusion locks and syncronization. Not that I didnt use it when I was
    writing win apps in C++, but it really wasnt the right way to solve the
    problem :)

    Re: sockets -- yes they are byte streams at the low level -- all IO is --
    even disk -- you can wrap them in a stream reader if you want a string/line
    oriented interface, or make your own reader to de-block the data any way you
    want :)
     
  19. colin

    colin Guest

    ah ok I guesed it was something a bit more complicated.
    done that.
    erm easy if you know how I gues, I didnt come accross events so far,
    im using the workerthread things dragged from the toolbox so I kind bypased
    the constructor details.

    I used System.Threading.AutoResetEvent, to wait for the queue to become non
    empty,
    wich is actually easy enough.
    but it sounds like these event handlers you mention before do the equivalent
    of
    what the old win32 messages do wich is what I was looking for,
    but just with the strong type control thats needed ?

    windows messages wasnt realy a lazy solution as they were the underlying
    fabric of mfc.
    they were not only used to talk from task to task but mostly within the same
    task.
    although there was the problem of ensuring what was responsible for the
    allocated memory
    wich c# takes care of.

    I find it hard to beleive the functionality of sscanf isnt in c#
    I found loads of articles of people trying to convert c++ code into c#
    and loads of people trying to answer but getting tied into knots with saying
    you just use
    regex or tryparse without realy knowing how to exactly,
    but these are either rather low level or overkill.
    trying to parse my variables wich are seperated with a variety of delimiters
    for clarity
    is irritating without an equivalent.

    I tried to make a function template but it wouldnt let me access the right
    TryPars member from the template. So i have one for each type. it works
    anyway.

    thanks
    Colin =^.^=
     
  20. John Barrett

    John Barrett Guest

    To create an event

    1. create a "public delegate <method prototype>;" outside the class that
    will have the event... inside the class -- add a "public event <delegate
    method name> <event name>;"

    2. any place in the class where you want to fire the event, add "if (<event
    name> != null) <event name>(delegate params);"

    3. in any class that you want to handle the event, add "object.<event name>
    += handler;"... the IDE will create the handler stub for you if you hit tab
    twice after the +=

    the delegate defines the parameters that will be passed, the event statement
    defines the variable that will contain the list of handlers to be invoked
    when the event is fired. Since delegates are typed, you can pass any
    number/type of parameters that you like -- no limitations :)

    Re sccanf == ya -- that doesnt really fit into the java/c# paradigm --
    you've probably already found this or something similar -- but here is a
    regex based sccanf for c#

    http://www.codeproject.com/csharp/CsScanf.asp
     
Ask a Question
Want to reply to this thread or ask your own question?
You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.
Electronics Point Logo
Continue to site
Quote of the day

-