μblog: engineering from the trenches

While looking for ways to escape muti-variate calculus purgatory in the final weeks of the semester, I came across Open Math Text.  These are a collection of math books (in PDF and LaTeX) that are openly available for distribution and are aimed at general scholars. A quick look at the collection will show that most of the books are authored by Dr. David Santos, a professor a the Community College of Philadelphia.  It seems that he has written and made available more books, in multiple languages, than the number of scholarly papers that most researchers publish at full universities.

While looking at his personal page, I found another open textbook collection called Textbook Revolution. The obvious downside is that these publications may not go through the same levels of review as textbooks printed at conventional publishers, however, it is nice to know that there is a group of people actively working to make affordable textbooks available.

del.icio.us | digg

As I have mentioned, the semester is winding down and projects are piling up. To help move things along, I have written a pair of documents that outline how to determine the conductivity of a plane (and volume) with a conducting disk (and sphere) of arbitrary size in the middle. I solved Laplace’s equations in both polar and spherical coordinate systems, then used boundary conditions to determine the electric potential and then determined the ratio of applied field to current density to determine the conductivity in the presence of the suspended object. I have checked these early drafts over a few times, however, there may still be some mistakes remaining, so please be warned. Also, feel free to post questions and I will make an attempt to answer them.

( disk-efield.pdf )

( sphere-efield.pdf )

P.S. The photo-op was staged during my last vacation, I would never use Classical Electrodynamics as a coaster.

del.icio.us | digg

The semester is winding down, which means there are a lot of deadlines piling up. One of my deadlines involves re-deriving some work by Hugo Fricke regarding the electrical properties of suspensions of conducting spheres in a conducting medium. Fricke was one of the pioneers of radio-therapy and was one of the first individuals to postulate that blood cells had membranes (instead of being homogeneous solids). He did this through electrical interrogation of blood alone without using any optical techniques. I am posting my step-by-step derivations for the electric potential inside and around a single conducting sphere in a conducting medium with regards to electrostatics. I solve Laplace’s equation using separation of variables in a spherical coordinate system. Hopefully I didn’t make many errors and the rest of the derivation relating total cell conductivity and capacity will follow.

( sphere-efield.pdf ) (Image is from GNU)

del.icio.us | digg

As promised before, I have finally worked through the majority of this paper, enough to give a brief introduction and discussion.

The key point of this paper is to demonstrate the importance of statistical analysis and its applications to determining information generation and transmission capacity. The measure H, or entropy, can be thought of as the amount of variance, or uncertainty, in a communication system. This leads us to define the theoretical capacity of a communication system given the known statistical properties of its constituents as well as apply analysis to practical systems.

The concept of information entropy deals with the uncertainty in the expected value of this information. Although it is rooted in statistical mechanics, it can be seen that highly predictable information has low variance, and therefore lower entropy, as compared to more random information. From this measure of information entropy, we can determine the necessary number of bits to efficiently encode this information, or to put it another way, how many symbols we can transmit per bit (assuming digital communication medium). Although the case of uniform probability distribution for all information symbols is easiest to analyze and leads to highest entropy, most practical applications have particular statistical distributions for symbol/information generation. Shannon goes to lengths to demonstrate this with the English language noting that selection of letters, or even words, is highly structured and far from random. This structure is a measure of redundancy of information, so that if I typ like ths, you cn stil undersnd me. (Spammers have been rediscovering this fact for years.)

Once the information entropy for all of the circuits involved in the communication system are determined, the channel capacity can be determined in the form of symbols per second given a finite certainty and a raw channel bit-rate. Shannon gives a fine example of a digital channel operating at 1000bits/s with a 1% error rate leading to an effective bit rate of ~919bits/s to account for error detection. Some communication system examples are given which I will not discuss in depth, however, I will try to reiterate the important steps in efficient communication design. Although Shannon gives a mathematical formulation for determining the theoretical limit for channel throughput, it is up to the designer to realize create a system which comes close to the limit. To do this, it is imperative to know the statistical properties of all of the sub-systems involved and the noise that may be present, and only then can efficiency be achieved.

The paper is by far more in-depth than this introduction and the math is not too hard, if anything, it is worth a look-over for some commentary on the statistical nature of the English language. As always, feel free to post a comment to discuss something about the paper, add something, or correct a mistake I have made. As a small bonus, I am adding Shannons’ patent for PCM-encoded voice/telephone service for those who like to read those types of things.

( 1948shannon-a-mathematical-theory-of-communication.pdf )
( 1946shannon-communication-syste-memploying-pulse-code-modulation-patent.pdf )

del.icio.us | digg

I have been working on spectral analysis of time series data and I am finding that (duh!) it is often easier to use someone else’s wheel than to invent a new one myself. That is, if their wheel has all of the necessary features and meets your performance criteria. I have been working with multi-taper spectral estimates and have found that the Chronux code (MATLAB scripts), available from Partha Mitra’s lab, to be fairly easy to use and effective for some continuous time-series analysis. The code also handles point process analysis, however, I don’t use that feature much.

The main reason that I like to use the multi-taper methods based on Slepian tapers, is that each taper gives a independent measurement of the signal spectra which then allows for computation of variances and therefore confidence intervals. This gives a certainty measurement to the analysis which can either make a strong case for its significance or will identify deficiencies in the data. Basically, these people put some years into writing this code and if you need to do coherence or spectral analysis in MATLAB, might as well give it a try.

del.icio.us | digg

Most scholarly articles that have some substance have a reference section at the end. The purpose of this section is to support scientific statements and to aid readers who would like to learn more about some of the article’s sub-topics. In rare occasions, authors use the reference section to show their scholastic diligence by citing some very old  or obscure sources. An example would be to include Laplace’s or Poisson’s equations in your paper, where relevant to electromagnetics, and then cite Maxwell’s Treatise on Electricity and Magnetism from the 1870s.  These are basic electromagnetics equations that are part of standard physics and engineering textbooks, which should be cited instead of Maxwell’s works. The rationale is that modern books are much easier to locate and are much easier to read. On the other hand, a good reason to cite Maxwell in a paper would be to discuss some particular part that is either unavailable or is contrasted to modern literature.

The purpose of this post is to present the best selling mathematical textbook in known history: Euclid’s Elements of Geometry. This contains a reproduction of the original Greek text along side a translation to modern English. Much of the contents of this tome are covered in typical high-school curricula, however, if you want to bolster your appearance of scholarship, this is probably one of the oldest works that anyone in the sciences will ever cite.

( elements.pdf )

del.icio.us | digg

Adding references, or data from, publications that are over a hundred years old seems to be a popular trend among scientific presentations these days. Sometimes it is to give a false sense of scholarship, however, it is often to remind us that some ‘new’ scientific breakthroughs may be simple re-interpretations of old discoveries. I try to note the references and look them up, when time permits. Here is the first paragraph from the preface of ‘Epilepsy and Its Treatments‘ (1904) by Spratling:

 The great progress made in the knowledge of epilepsy and its treatments during the past decade and a half, and in fact that no complete treatise on the subject has appeared in the United States since Echeverria’s work was published thirty-three years ago, was the chief reason that lead to the preparation of this volume.

With the exception of Manuel Echeverria (On Epilepsy: Anatomo-Pathological and Clinical Notes (1870)), the sentence can still be used in a modern book/review of Epilepsy without much alteration. The reason that the 1904 book was cited was to show that, a hundred years ago, physicians were aware that, on very rare occasions, were non-clinical. For example, it was noted that verbal interactions were sometimes enough to bring people out of seizure, something that researchers who seek alternative epilepsy treatments are rediscovering. (On a slight side note, there is an interesting personal account by Feydor Dostoyevsky starting at the bottom of page 466 where he links a pre-seizure state to a state of mental enlightenment.)

This long winded introduction was to present a pair of review articles from the early 1900s that covered what the authors thought were the highest achievements in physics and applied math of the previous century.

1905barus-the-progress-of-phsyics-in-the-nineteenth-century.pdf

1900woodward-the-centurys-progress-in-applied-mathematics.pdf

The image is from the IEEE and is of the Georgetown, CO steam/hydro powerplant.

del.icio.us | digg

This very small introduction to the Central Limit Theorem is probably something worthwhile before the Shannon paper. The main point is that as we take more and more samples from a random variable, with a fixed mean and variance, the samples approach a normal (Gaussian) distribution. That is, irregardless of the distribution of the random variable, if it meets the criteria, it will behave like a normal ly distributed random variable in the limiting case. The typical application engineering application of this theorem is making the assumption that some measured quantity is normally distributed and use that assumption to define things like confidence limits and so forth. The requirements for this assumption are that the process is second-order stationary, meaning the mean and variance do not change in the window of observation, and that the number of samples is approaching infinity. The requirement for a large number of samples can sometimes be loosened since the residual differences between the sample distribution and a normal distribution can sometimes be determined. The requirement for a stationary process cannot. For example, it would be foolish to apply Gaussian statistics to a random-walk (Brownian motion).

The key message is that the normal/Gaussian assumption is typically a good one, as long as the statistical nature of the random variable under investigation is constant through the period of observation and the number of samples is large.

( sec_4_f.pdf ) ( Image is from Wikipedia )

del.icio.us | digg

When trying to do parameter estimation given a set of data, there are typically two approaches: least squares estimation and maximum likelihood estimation. In both cases, a model must be constructed where the first case tries to fit the data to the model by minimizing residual errors while the second method tries to estimate the probability density function associated with the collected data and thereby determine the parameters. While trying to make sense of things, I found this tutorial on maximum likelihood estimation by In Jae Myung of Ohio State University to be very helpful as it provided a description as well as MATLAB code examples. (He also also publishes a list of books currently on his bookshelf!)

Update: Here is a link to a paper outlining R.A. Fishers arrival at the concept of maximum likelihood. An interesting thing to note here is that given a likelihood function P, log(P) is often maximized, yielding maximum likelihood because the function would have to be differentiated and given that the probability distribution of many naturally occurring events is Gaussian, differentiating the logarithm of such a probability density function just makes more sense.

del.icio.us | digg

While looking up some papers on statistics I came across this freely-available copy of William Gosset’s March 1908 paper, almost a year before the its United States copyright expires and it becomes public domain. The paper is a very nice transcription of the original paper (available through JSTOR) which makes it much easier to read.

The point of the paper relates a statistical confidence limit and a set of bounds within which the mean of a set of measurements must lie. To be more specific, the paper addresses measurements whose distribution is Gaussian and goes on to specify how small a error bound you can place around the mean given a finite number of experiments with a finite certainty, say 95%. Furthermore, the usefulness of the paper comes from the fact that the limit is examined with a decreasing number of observations. This is useful since certain experiments cannot be performed many times and we need to be able to say how certain we are of the mean of the observed Gaussian random variable given the finite number of data points.

In a small side note, the p < 0.05 (or >95% “certainty”) is often considered to be a “good” value, but it may seem somewhat arbitrary. There are some people who attribute this to Karl Fisher, more specifically, his publication titled Statistical Tables for Agricultural, Biological and Medical Research (3.3MB). For those looking for Fisher’s historical papers, they can be found at the University of Adelaide.

( 1908student-the-probable-error-of-a-mean.pdf )

del.icio.us | digg