Connect with us

What Nyquist Didn't Say

Discussion in 'Electronic Design' started by Tim Wescott, Sep 29, 2006.

Scroll to continue with content
  1. Tim Wescott

    Tim Wescott Guest

    I've seen a lot of posts over the last year or so that indicate a lack
    of understanding of the implications of the Nyquist theory, and just
    where the Nyquist rate fits into the design of sampled systems.

    So I decided to write a short little article to make it all clear.

    It's a little longer than 'short', and it took me way longer than I
    thought it would, but at least it's done and hopefully it's clear.

    You can see it at
    http://www.wescottdesign.com/articles/Sampling/sampling.html.

    If you're new to this stuff, I hope it helps. If you're an expert and
    you have the time, please feel free to read it and send me comments or
    post them here.

    --

    Tim Wescott
    Wescott Design Services
    http://www.wescottdesign.com

    Posting from Google? See http://cfaj.freeshell.org/google/

    "Applied Control Theory for Embedded Systems" came out in April.
    See details at http://www.wescottdesign.com/actfes/actfes.html
     
  2. Joerg

    Joerg Guest

    Hello Tim,

    Very nice. Should be distributed to universities so the kids learn some
    real stuff.

    Re "3.2 Nyquist and Signal Content": If Nyquist would have listened to
    some of today's content (digital radio etc.) he'd have said that it
    ain't worth sampling it :)

    I like the wording "line in the sand". Isn't that how Archimedes started
    studying his circles? They didn't need any white board with the smelly
    marker pens.
     
  3. John Larkin

    John Larkin Guest


    Pretty good. My only quibble is the various statements about what
    Nyquist said and didn't say.

    http://en.wikipedia.org/wiki/Nyquist-Shannon_sampling_theorem#Historical_background


    "Exactly how, when, or why Nyquist had his name attached to the
    sampling theorem remains obscure. The first known use of the term
    Nyquist sampling theorem is in a 1965 book[4]. It had been called the
    Shannon Sampling Theorem as early as 1954[5], but also just the
    sampling theorem by several other books in the early 1950s."

    It was actually Shannon (among others) that did the sampling theorem;
    Nyquist made an observation. Your bibliography doesn't cite either of
    them. It's probably correct to use "Nyquist rate" but not "Nyquist
    theorem."

    John
     
  4. Joerg

    Joerg Guest

    Hello John,
    Nyquist published his paper about the minimum required sample rate in
    1928. Shannon was a kid of 12 years back then. The paper wasn't about
    ADCs or sampling in today's sense but about how many pulses per second
    could be passed through a telegraph channel of a given bandwidth.
     
  5. Jim Stewart

    Jim Stewart Guest

    "2.1 Aliasing

    By ignoring anything that goes on between samples the sampling process
    throws away information about the original signal. This information
    loss must be taken into account during system design."

    This seems like something of an oversimplification. If the orginal
    signal is naturally or otherwise bandwidth-limited to well below
    2x the sample rate, there may not be any useful information available
    to throw away and the loss may not have to be taken into account
    during system design.
     
  6. Tim Wescott

    Tim Wescott Guest

    Thanks John, that's a good point. I'll probably leave the titles intact
    because the paper is a reaction to all the posts that have the words
    "Nyquist says" followed by something _wrong_ -- but I should put in a
    disclaimer, or something.

    I'm going to go off and do some web searching; in the mean time do you
    have any URLs that point to the seminal papers?

    --

    Tim Wescott
    Wescott Design Services
    http://www.wescottdesign.com

    Posting from Google? See http://cfaj.freeshell.org/google/

    "Applied Control Theory for Embedded Systems" came out in April.
    See details at http://www.wescottdesign.com/actfes/actfes.html
     
  7. Tim Wescott

    Tim Wescott Guest

    I can squirm out of that objection:

    If you have the continuous-time signal, then you _know_ there's nothing
    of note above Fs/2. If you don't have the continuous-time signal, then
    you _can't_ know there's nothing of note above Fs/2, unless the sampled
    signal train comes with a Certificate of Limited Bandwidth.

    Later on in the article I talk about signals that are, indeed,
    sufficiently bandlimited by their nature, and the fact that you probably
    don't want to do any explicit anti-alias filtering in such a case.

    --

    Tim Wescott
    Wescott Design Services
    http://www.wescottdesign.com

    Posting from Google? See http://cfaj.freeshell.org/google/

    "Applied Control Theory for Embedded Systems" came out in April.
    See details at http://www.wescottdesign.com/actfes/actfes.html
     
  8. John Larkin

    John Larkin Guest

    The wiki article has links to Nyquist's 1928 paper and to Shannon's
    1949 paper. Few EEs that I've met have ever read either of them, and
    have absorbed "the Nyquist theorem" mostly by hearsay, which I guess
    is your point.

    John
     
  9. John Larkin

    John Larkin Guest

    Don't you go knocking whiteboards. A big board with a nice fresh set
    of markers will multiply my IQ by about 1.3 or so. Sand doesn't work
    nearly as well.

    Oh well, back to the drawing board, literally.

    John
     
  10. Joerg

    Joerg Guest

    Hello Tim,

    This one could be a start, written by my old communications theory
    professor (he was actually one of the really nice profs):

    http://www.ee.technion.ac.il/courses/044130/00755459.pdf
     
  11. I've read Shannon's paper, thorougly. At least, these: The Bell
    System Technical Journal, Vol. 27, pp. 379–423, 623–656, July,
    October, 1948. Note that this is NOT 1949. But it was, in fact, what
    made me understand Boltzmann much better than before and allowed me to
    better access the underlying meaning of macro concepts such as
    temperature and entropy (which have no micro-meaning.)

    Hamming, Shannon, and Golay all worked together in the same place, if
    I recall, around that time. Marcel J. E. Golay, 1949, and a little
    (short) paper called, "Notes on Digital Coding" which came out in
    1949. (I think he was pushed into it by Hamming and Shannon.)

    Jon
     
  12. John Herman

    John Herman Guest

    In the first sentence of your papere, you say that if the signal is band
    limited to fo or less, then a sample frequency of 2fo or more is adequate to
    contain completely all the information necessary to recreate the signal. My
    understanding from water cooler conservation is that the signal bandwidth must
    be strictly less than fo for a sample frequency of 2fo and that the signal
    must be infinite in extent to allow perfect reconstruction.

    IMHO. I could be wrong. I have never seen a formal proof just some
    handwaving.
     
  13. Exccept it was Shannon that said it...
     
  14. Some more info and a correction.

    First, the correction.

    Golay didn't actually with Shannon at Bell labs. His first paper, the
    one I mentioned called "Notes on Digital Coding" was actually
    published in the Correspondence section of Proc. I.R.E., 37, 657
    (1949) was written while he was at the Signal Corps Engineering
    Laboratories in Fort Monmoth, N.J.

    Now, the additional info.

    Golay's 1949 paper is supplemented by two more papers he wrote:
    "Binary Coding" I.R.E. Trans. Inform. Theory, PGIT-4, 23-28 (1954) --
    written also from Fort Monmoth, NJ; and "Notes on the Penny-Weighing
    Problem, Lossless Symbol Coding with Nonprimes, etc.," I.R.E Trans.
    Inform. Theory, IT-4, 103-109 (1958) -- written from Philco
    Corporation when in Philadelphia, Pa.

    Twelve of Shannon's papers (1948 through 1967) are conveniently
    collected in the anthology, "Key Papers in the Development of
    Information Theory," edited by David Slepian (IEEE Press, 1974.)

    Shannon referenced Golay's 1949 paper in the book "The Mathematical
    Theory of Communication" (written with Warren Weaver, Univ. Illinois
    Press, 1949.) This book contains a slightly rewritten version of
    Shannon's first 1948 papers together with a popular-level paper by
    Weaver.

    Shannon describes the Hamming-7 code in his 1948 papers in section 17,
    attributed to Hamming there, but since there is no reference to a
    specific paper by Hamming I suspect this reference must have been via
    personal communication with Hamming. (Golay also refers to the
    Hamming-7 code in Shannon's first paper.)

    The first paper by Hamming is "Error Detecting and Error Correcting
    Codes" Bell System Tech. J., 29, 147-160 (1950.) Note this is
    actually _after_ Shannon's reference to Hamming's code. The
    anthology, "Algebraic Coding Theory: History and Development," edited
    by Ian F. Blake (Dowden, Hutchinson & Ross, 1973) includes this paper.

    Blake says in his introduction to the first 9 papers in his anthology:

    "The first nontrivial example of an error-correcting code appears,
    appropriately enough, in the classical paper of Shannon in 1948. This
    code would today be called the (7,4) Hamming code, containing 16 = 2^4
    codewords of length 7, and its construction was credited to Hamming by
    Shannon. Golay gives a construction that generalizes this code over
    GF (p), p a prime number, of length (p^n -1)/(p - 1) for some positive
    integer n. Hamming also obtained the same generalization of his
    example of codes of length (2^n - 1) over GF(2) and investigates their
    structure and decoding in some depth. The codes of both Golay and
    Hamming are now designated as Hamming codes. The interest of Golay
    was in perfect codes, which have also been called lossless, or
    close-packed, codes. Since he mentions the binary repetition codes
    and gives explicit constructions for his remarkable (23,11) binary and
    (11,6) ternary codes, it is not stretching a point to say that in the
    first paper written specifically on error-correcting codes, a paper
    that occupied, in its entirety, only half a journal page, Golay found
    essentially all the linear perfect codes which are known today. ...
    The multiple error-correcting perfect codes of Golay, now called Golay
    codes, have inspired enough papers to fill a separate volume."

    By the way, the Hamming-7 code can be used to generate the E7 root
    lattice, which corresponds to the E7 Lie algebra. And the Hamming-8
    code similarly generates the E8 root lattice, corresponding to the E8
    Lie algebra. Heterotic superstring theory has an E8 x E8 symmetry
    which is needed for anomaly cancellation. It is nifty that the very
    first error-correcting codes of Golay and Hamming play such a profound
    role in modern superstring theory.

    An interesting supplemental work is from J. H. Conway and N. J. A.
    Sloane, "Sphere Packings, Lattices and Groups" from Springer-Verlag,
    1988. But probably the best ever book on algebraic coding theory is:
    "The Theory of Error-correcting Codes" by F. J. MacWilliams and N. J.
    A. Sloane, North-Holland Publishing Co., 1977.

    Jon
     
  15. jasen

    jasen Guest

    No expert, but figure 14 could possibly be improved by explicitly showing
    the rising edge of the pulse, maybe start the time axis at -0.5 or start
    the pulse at +0.5

    The "contact us" link appears to be broken too.
     
  16. xray

    xray Guest

    You guys are good.

    I doubt I'll ever read all that myself, but I do appreciate the work and
    motivation it took to take the time to care and find it all.

    I'll only say that I hope all of you all keep your enthuiasm to dig,
    find the best answer, and share.

    It's why I hang out here, read, and snipe occasionally.

    As Martha said, even back before she became a capitalistic criminal,
    "It's a good thing."
     
  17. Rune Allnor

    Rune Allnor Guest

    Tim Wescott skrev:
    Tim,

    I really like your style of writing. Accessible, and yet with no hint
    at the "for dummies" craze.

    I hadn't really thought of the Signal-to-Aliased-Energy ratio as a
    metric for signal quality before. It will be a lot easier to discuss
    the
    quality of sampled systems with that as a tool. Very good
    presentation of the both the concept and the effect. I whish
    it was me who thought of that sequence of figures 6-9...
    oh well.

    A couple of very subjective comments:

    - There is something about the typography of the page that annoys me.
    I can't really see if there is an open line between paragraphs inside a

    section; if there is, you may want to make it clearer

    - The footnote indicators are WAY too small to be useful, or even seen.


    - I can't see why you need the maths paragraph ("To understand
    aliasing...", eqs. (2),(3)) at all in section 2.1. I think that what
    you
    say there is covered by the plain-text parts of the section. Lay-men
    might find that maths paragraph scary, throwing them off your article,
    while technicians know it already.

    - Similar comments apply for eq. (4). I can't see that it is strictly
    necessary, the explanation is in the text anyway.

    - Eqs. (5)-(7) seem to be necessary, but are the only big equations
    in the article. Hmm... that makes me wonder...

    As far as I can tell, you are 99.9% at the point where this is an
    article a non-engineer layman can read and understand. If you
    find a way to get rid of equations 1-7, maybe even eq. 8, you
    ought to be there.

    Impressive work!

    Rune
     
  18. Guest

    Very good. I would change this phrase:
    "This will allow you to use a more open anti-aliasing filter."
    By "more open", I gather you mean a larger transition ratio, which
    lowers the Q of the resonators.

    Note that the Bessel filter will ring at higher orders. I don't have my
    copy of Zverev handy, but I think the Bessel rings at 4th order and
    higher. The Gaussian filter doesn't ring at any order. The key is to
    look at the impulse response of the filter. If it ever goes negative,
    then filter will ring.

    You might want to go into the inverse sinc response requirements in the
    recontruction filter.
     
  19. Since understanding the formula _notation_ is not essential for
    understanding most of the rest of the text, I would suggest moving the
    formulas into separate boxes and moving these boxes out of the direct
    text flow e.g. into a box in the right margin of the paper.
    Definitively !

    The problem with quite a few text dealing with sampling is that they
    are written by mathematicians for the mathematicians. However, these
    days, most people using various DSP algorithms are programmers, not
    mathematicians, thus the terse mathematical notation can be hard to
    understand to them, especially without too much numerical analysis
    background.

    Instead of using the terse mathematical notation, it might be more
    productive for most readers to publish e.g. a Fortran/Algol/C
    algorithm.

    Paul
     
  20. Mike Monett

    Mike Monett Guest

    Are you sure about that? Here is a 9th order 1 MHz Bessel for LTspice.
    (Save as a CKT file)

    * UTS Mike Monett
    * Converted From Micro Cap Source file to LTspice
    *
    C1 0 1 248.3PF
    C2 0 2 1200.0PF
    C3 0 3 2007.3PF
    C4 0 4 2749.9PF
    C5 0 Vout 7209.4PF
    L1 1 2 1.8UH
    L2 2 3 4.1UH
    L3 3 4 5.9UH
    L4 4 Vout 8.6UH
    R1 Vin 1 50
    R5 0 Vout 50
    V1 Vin 0 DC 0 PULSE (0 1 0 0 0 2.5e-006 5e-006)
    ..TRAN 1e-008 10u 0 1n UIC
    ..PRINT TRAN V(VOUT) V(VIN)
    ..PLOT TRAN V(VOUT) V(VIN)
    ..PROBE
    ..END
    ;$SpiceType=SPICE3

    It doesn't ring.

    Regards,

    Mike Monett

    Antiviral, Antibacterial Silver Solution:
    http://silversol.freewebpage.org/index.htm
    SPICE Analysis of Crystal Oscillators:
    http://silversol.freewebpage.org/spice/xtal/clapp.htm
    Noise-Rejecting Wideband Sampler:
    http://www3.sympatico.ca/add.automation/sampler/intro.htm
     
Ask a Question
Want to reply to this thread or ask your own question?
You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.
Electronics Point Logo
Continue to site
Quote of the day

-