(See Part 1, Part 2, and Part 3. Also I am going to suspend the limit of three comments per post for this series of posts because it is a topic that benefits from back and forth discussions.)

Even if we decide to treat the microscopic and macroscopic worlds as separate and governed by different laws, they is one place where the two world collide that we cannot ignore. Recall that we said that in the quantum world, many results do not come into existence until they are measured. Any contact at all of a quantum superposition of states with a macroscopic object, however small, can cause the collapse of the wave function. But in order for it to be useful to us, we need to know what the final result was, and that means we need a measurement involving a measuring device whose results we can see, such as a detector like a fluorescent screen, photometer, bubble chamber, geiger counter, and so on. So when we measure (say) the spin or location of an electron, we unavoidably have an interaction of an object that belongs to the classical world (the detector) with an object that belongs to the quantum world and this leads to what is called the measurement problem.

To understand the measurement problem, recall that we start with a quantum system that is prepared so that a particle (say an electron or photon) is created such that we cannot predict which state (spin up or spin down) it will be found in upon measurement. We describe the wave function of this particle as being in a superposition of two states, one spin up and one spin down. (Such a superposition of states is said to be coherent.) This superposition will continue to exist as long as the particle does not interact with anything that can be considered macroscopic, however small. When it does, the wave function is said to abruptly shift from being in a superposition of the two states to just one of the states. (This process is referred to as decoherence.) We can’t predict with certainty which state it will collapse into but if we know the initial wave function (say because it is a solution of the Schrodinger equation that we are able to obtain), we can predict the probability of collapsing into each one.

So an act of measurement is when a particle in a quantum state whose wave function is a superposition of states meets a macroscopic object (a detector that serves as the observer) that causes the wave function to suddenly collapse into one of its states. This collapse is sometimes referred to as a quantum jump. The measurement problem is that we do not know what exactly happens during this ‘jump’, even though quantum theory seems to require it to happen all the time, whenever a measurement occurs. The mysterious and unknown nature of this highly ubiquitous process is distasteful to pretty much everyone. This was true not just for Einstein but also for Ervin Schrodinger ,who gave us his eponymous equation which we use to calculate the wave function. He too hated the idea that the wave function abruptly jumped from one state to another and famously said, “If we have to go on with these damned quantum jumps, then I’m sorry that I ever got involved.” We tolerate it because quantum mechanics works so well and nothing better has been found that has been successful in convincing most physicists to adopt it.

Not that there have not been attempts to find alternatives. There have been strenuous efforts to remove this ugly feature of mysterious quantum jumps upon measurement by getting rid of the role for the observer (detector) that causes it. Some of these efforts come under the heading of hidden variable theories that introduced additional variables that, contrary to standard quantum theory, determine the outcome of the measurement before the measurement, even though we may not know precisely what those variables are.

The famous mathematician John von Neumann purported in 1932 to have proved on general grounds that such hidden variables were impossible within the framework of quantum theory and because of his eminence, that carried a lot of weight and put a real damper on any attempts in that direction. But while there were some private grumblings that von Neumann’s ‘impossibility proof’ may contain a faulty assumption that invalidated his strong conclusion, it was John Bell who in 1966 published a paper showing that it was flawed. Hidden variable were possible at the price of the theories being non-local. Bell said that he was inspired to investigate von Neumann’s claim when he read a paper published by David Bohm in 1952 (following up an idea by Louis de Broglie in 1927) which explicitly contradicted von Neumann’s assertion of impossibility by constructing a model that was supposedly impossible. In this model an electron (say) in addition to its wave function, is guided by an additional ‘pilot wave’ (now called the de Broglie-Bohm pilot wave) that makes its motion completely determined, like that of a classical particle, while still giving us results that are consistent with quantum mechanical experiments. One consequence of doing so, as Bell showed in 1964, is that this theory, like all hidden variable theories (a term that Bell dislikes), is necessarily non-local, which is also somewhat distasteful.

But there are other theories. In one model published in 1986 by Ghirardi, Rimini, and Weber, the wave function collapse is not due to any interaction with a detector. Instead, the wave function is randomly and spontaneously and repeatedly collapsing on its own into a single observable state, then evolving quantum mechanically again so that it could be in a superposition, then collapsing again, and so on. The collapse is built into the system and not caused by any external detector. It is influenced by the size of the difference between the two superposed states. So for a small system like an electron in spin up and spin down states, the time between these spontaneous collapses could be of the order of millions of years whereas for a macroscopic object containing say 10²⁰ particles or more, the time between collapses is of the order of 10^-5 seconds or less. So macroscopic objects keep collapsing after infinitesimally small amounts of time so it looks like they behave classically and are never in superposed quantum states, while microscopic objects like electrons can have a large amount of time between collapses, which is why we observe quantum effects only in microscopic systems.

Perhaps the most imaginative and attention-grabbing of the alternatives is the many worlds interpretation (MWI) first proposed by Hugh Everett in 1967. In this model, while standard quantum mechanics is the accepted theory, there are no quantum jumps. Instead at any stage in which an electron (say) was in a superposition of two states and is then detected to be in one, it is not the case that the wave function abruptly collapsed or jumped into that state. What is said to have happened is that the universe split into two with each universe having one of the two results. So when we say that a measurement gave us spin up, what happened is that we find ourselves in a universe where the spin was up, but that there is now a parallel universe in which the parallel ‘we’ found the electron to be spin down. But since the two universes do not communicate with each other, we think that the wave function collapsed to just the one observable state that we see.

The many-worlds interpretation implies that there are many parallel, non-interacting worlds. It is one of a number of multiverse hypotheses in physics and philosophy. MWI views time as a many-branched tree, wherein every possible quantum outcome is realized. This is intended to resolve the measurement problem and thus some paradoxes of quantum theory, such as Wigner’s friend,  the EPR paradox,  and Schrödinger’s cat, since every possible outcome of a quantum event exists in its own world.

As you can see, some of these theories are quite imaginative. This is because when an important problem in physics is proving to be intractable (and the measurement problem definitely fits that bill), people are willing to entertain ideas that are quite far from the orthodox. This is common in science. This ambiguous state of affairs continues until there is a preponderance of evidence that shifts the consensus firmly in one direction.

There is no such consensus now about what to do about quantum measurements and the associated jumps. What there is a consensus about is that whatever final theory is adopted must incorporate the results of standard quantum theory into its framework because those results are so robust and extensive. All these alternatives allow for that, if not exhaustively, at least minimally to the extent that they have been investigated. But there is no convincing evidentiary basis for any of them. As a result, where one ends up depends on personal preference and all of these theories have their adherents and one cannot say that any of them are wrong. Standard quantum theory with inexplicable jumps and nothing more is probably the most popular but hidden variable theories and the MWI have many supporters. Sean Carroll is one advocate of the MWI and he addresses some of the common misconceptions about it.

In the next and final post, I will get to another imaginative alternative, the one that triggered this whole series of posts, and that is the possible role of consciousness in measurement.