Fact and Fiction

Thoughts about a funny old world, and what is real, and what is not. Comments are welcome, but please keep them on topic.

Sunday, July 23, 2006


I will be on holiday in the south-west of England for the next 2 weeks, and I won't be able to access the internet, so any comments will remain unanswered until I return.

Update: I extended my holiday to 3 weeks because I couldn't face going back to this! Unfortunately, the weather didn't arrange a similar extension. I spent a lot of time in the West Penwith area of Cornwall, with a brief interlude at the Big Green Gathering at Fernhill Farm in Somerset. Due to the walking/cycling that I did whilst away I now weigh several pounds less than before my holiday, and I even look slightly healthy due to the sun/wind tan that I acquired. The peace and tranquillity once you are off the beaten track in parts of West Penwith is very relaxing, and I can recommend the area for "chilling out", but I won't tell you precisely where I stayed.

Friday, July 21, 2006

Pioneering research - a risk worth taking

I have just received another batch of books from Amazon, and I have started to read Pioneering Research: A Risk Worth Taking by Donald Braben. Here is a choice quote that appears in the introduction:

"To summarise my story therefore: The greatest long-term risks facing humanity will not come from such apocalyptic threats as terrible weapons of mass destruction, prolonged global war, devastating disease or famine or extinction by a huge wayward meteor. Rather they will come from the debilitating attrition caused by the rising tides of bureaucracy and control. These trends are steadily strangling human ingenuity and undermining our very ability to cope."

What a hook! Now I have to read the rest of the book (NB: I haven't yet done so). This description of bureaucracy as a "strangler" succinctly describes my own experience of bureaucracy, so I can't resist making some comments about it here.

It seems to me that the spread of cheap computers and powerful software has had an unfortunate side-effect, where bureaucrats are now able to do much more than they could before the advent of computers. There are people with a certain type of mentality who love using computers. Whilst cheap computers were limited in their capabilities such people were limited to really geeky types (tech-geeks), who loved to control their little "computer universe" with very clever computer programs. Now the same amount of money buys you a computer that can effortlessly run Microsoft Office, so we have a new type of geek (admin-geek) who wears a suit, but who has basically the same mentality as the earlier tech-geeks. Meanwhile tech-geeks have become über-tech-geeks, but that's another story.

The admin-geeks love their spreadsheets, and they feel ever so good when they "capture data" to "populate the spreadsheet". They recognise (correctly) that they can create order out of chaos by building business models out of various MS Office documents, and they might even discover how to link these documents together so that they update each other automatically without data having to be reentered time and time again. Actually, my experience is that they rarely get this last bit right.

The problem is that these simplistic business models are to real-world businesses as simple Gaussian probabilities are to real-world statistical processes. In both cases the model is well-intentioned but naive. If you use a simple Gaussian probability model when the real-world actually has a longer-tailed distribution, then your model is going to be badly wrong out in the tails of the distribution. If you use a simplistic business model when the real-world business wants to behave in ways that differ from the model, then there is going to be dissent between the bureaucracy and those whom they seek to control.

It is easiest to build a business model that describes a tightly controlled business, which consists of "if this condition holds then do that action else do that other action" clauses mutually interacting with each other. You can readily imagine the admin-geeks gleefully constructing a business model along these lines.

Back to the main theme of pioneering research. This particular activity does not easily fit into the "if-then-else" approach to business modelling. Unfortunately, the effect of this simplistic approach to business modelling has the effect of strangling the very thing that is needed to make pioneering research flourish, i.e. the freedom to follow your investigations wherever they lead you. The continual desire by bureaucrats to measure what you are doing, so that they can keep track of it in their business model, causes interference that damages the very thing that they are trying to measure in the first place.

Are there any simple fixes to this problem? I think that the admin-geeks lack basic trust and respect for the people who they are measuring/controlling. All they appear to see are the cells in their spreadsheets, and the relationships between these cells, but they appear to forget that these cells have real-world counterparts. Exactly the same problem happens with tech-geeks who get their noses buried so deeply in their computer programs that they can't relate what they are doing to the real-world. So the fix to the problem (in both the admin-geek and tech-geek cases) is to not only look at the real-world indirectly through the distorting lens of a computer program, but also to try to interface directly with the real-world, and to develop a direct intuitive appreciation for what is going on. Unfortunately, this requires some effort from everyone, which seems unlikely because using a computer to do your thinking for you is (or seems to be) so easy.

I wonder what Pioneering Research: A Risk Worth Taking says about this. I must finish the book.

Update: I have now read the book, and everything it says is consistent with my own unpleasant experiences with bureaucrats who micro-manage research. There were places where the phraseology used in the book corresponds exactly to the ways that I have described various situations to my technical colleagues at work. I have therefore lent it onto one such coworker in order to spread the word, which will thus delay any further discussion about the book here.

Thursday, July 20, 2006

Everyone uses prior probabilities

There is a lot of rubbish about the Bayesian approach to inference being written in various blogs; to spare peoples' blushes I will not cite those blogs here. As I see it, the main error that people make is to assume that the Bayesian approach is guilty of being subjective because it uses prior probabilities, which seemingly have to be plucked out of thin air (i.e. subjectively).

Fortunately, or unfortunately, depending on your point of view, the Bayesian approach is not the only one that is subjective according to these criteria.

Bayes theorem allows joint probabilities to be split up into products of conditional probabilities and marginal probabilities. The simplest statement of Bayes theorem is

Pr(x, y) = Pr(x) Pr(yx) = Pr(y) Pr(xy)

where the marginal probabilities are defined as

Pr(x) ≡ ΣyPr(x, y) and Pr(y) ≡ ΣxPr(x, y)

This allows us to write the conditional probability Pr(xy) as

Pr(xy) = Pr(x)Pr(yx) / (ΣxPr(x)Pr(yx)) = Pr(x) Pr(yx) / Pr(y)

where the dummy x that is used inside the summation is different from the free x that occurs elsewhere.

So, in order to determine x given that you know y (i.e. Pr(xy)), all you need to know is how to determine y given that you know x (i.e. Pr(yx)) together with the prior probability of x (i.e. Pr(x)).

The criticism that the Bayesian approach is subjective arises from the Pr(x) term. Why should drawing an inference about x given y depend on this apparently subjective factor? If x is the value of a physical constant, and y is an experimental measurement of it, then why should the interpretation (i.e. Pr(xy)) of this measurement apparently be subjective?

Firstly, there is an extreme case that we must dispose of. If the experimental data actually measures the quantity of interest with zero error (e.g. y = x) then the choice of Pr(x) has no effect. I am not talking about this extreme case. I am talking about the more realistic case where the data contains only partial information about the quantity of interest, because it is subject to noise, or maybe because it is a lower-dimensional projection of a higher-dimensional quantity.

Do some dimensional analysis (this is valid whether x and y are continuous or discrete):

  1. Pr(x) and Pr(xy) both have the dimensionality of 1/x.
  2. Pr(y) and Pr(yx) both have the dimensionality of 1/y.

Thus in Pr(xy) = Pr(x) Pr(yx) / Pr(y) the 1/x dimensionality of Pr(xy) derives entirely from Pr(x), because Pr(yx) / Pr(y) is dimensionless.

You have to use something like the dimensional Pr(x) factor in order to construct Pr(xy). If this factor is not actually Pr(x) itself, perhaps because you don’t like to use apparently subjective quantities, then what else could it be? Because it has inverse linear dimensions it has to be physically like a density. What densities do we have lying around ready for use? If we imagine that x-space is composed of infinitesimally small x-cells, then the density of such cells has the required properties. How do we decide what a good choice for these cells might be? Do we make them all the same size when viewed in x-space, or exp(x)-space, or what? There is no uniquely obvious choice!

The problem of choosing a space in which to define the cell size, so that a density can be defined in order to give Pr(xy) its dimensions, is the same as the problem of defining the Bayesian prior Pr(x). This is the reason why the Bayesian approach is not the only one in which you define a prior probability. Actually, in all approaches you have to define a prior probability, but only Bayesians use the term "prior probability", so they are totally honest about what they are actually doing.

You could define a density-like quantity in terms of the frequency of visits to each point in the space, which gives a number-per-unit-cell (i.e. a density). If you have an underlying model for generating points in x-space, then in principle it is easy to generate this type of density, and this would indeed be an objective way of defining a density. However, this ducks the issue of where the underlying model came from in the first place. There is a bootstrapping process, where you need to impose a density-like quantities on spaces that have never been visited before, and for which there is no agreed upon underlying model.

In short, everyone faces the problem of defining prior probabilities, but only Bayesians call them prior probabilities. It is disingenuous to use prior probabilities as a stick to beat Bayesians with. Unless, of course, you are a masochist, because everyone uses prior probabilities.

Tuesday, July 11, 2006

In memoriam Syd Barrett, madcap genius

So. Farewell
Then Syd

You were
Famous for
Your big hit
You can ride
It if you

Shine on
You crazy