Tag Archives: randomness

Idea Soup

Luck

Four-leaf_clover

Some think they have it constantly at their side, like a well-trained puppy. Others crave and seek it. And yet others believe they have been shunned by it. Some put their love lives down to it, and many believe it has had a hand in guiding their careers, friendships, and finances. Of course, many know that it — luck — plays a crucial part in their fortunes at the poker table, roulette wheel or at the races. So what really is luck? Does it stem from within or does it envelope us like a benevolent (mostly) aether? And more importantly, how can more of us find some and tune it to our purposes? 

Carlin Flora over at Aeon presents an insightful analysis, with some rather simple answers. Oh, and you may wish to give away that rabbit’s foot.

From aeon:

In 1992, Archie Karas, then a waiter, headed out to Las Vegas. By 1995, he had turned $50 into $40 million, in what has become known as the biggest winning streak in gambling history. Most of us would call it an instance of great luck, or we might say of Archie himself: ‘What a lucky guy!’ The cold-hearted statistician would laugh at our superstious notions, and instead describe a series of chance processes that happened to work out for Karas. In the larger landscape where randomness reigns, anything can happen at any given casino. Calling its beneficiaries lucky is simply sticking a label on it after the fact.

To investigate luck is to take on one of the grandest of all questions: how can we explain what happens to us, and whether we will be winners, losers or somewhere in the middle at love, work, sports, gambling and life overall? As it turns out, new findings suggest that luck is not a phenomenon that appears exclusively in hindsight, like a hail storm on your wedding day. Nor is it an expression of our desire to see patterns where none exist, like a conviction that your yellow sweater is lucky. The concept of luck is not a myth.

Instead, the studies show, luck can be powered by past good or bad luck, personality and, in a meta-twist, even our own ideas and beliefs about luck itself. Lucky streaks are real, but they are the product of more than just blind fate. Our ideas about luck influence the way we behave in risky situations. We really can make our own luck, though we don’t like to think of ourselves as lucky – a descriptor that undermines other qualities, like talent and skill. Luck can be a force, but it’s one we interact with, shape and cultivate. Luck helps determine our fate here on Earth, even if you think its ultimate cause divine.

Luck is perspective and point of view: if a secular man happened to survive because he took a meeting outside his office at the World Trade Center on the morning of 11 September 2001, he might simply acknowledge random chance in life without assigning a deeper meaning. A Hindu might conclude he had good karma. A Christian might say God was watching out for him so that he could fulfil a special destiny in His service. The mystic could insist he was born under lucky stars, as others are born with green eyes.

Traditionally, the Chinese think luck is an inner trait, like intelligence or upbeat mood, notes Maia Young, a management expert at the University of California, Los Angeles. ‘My mom always used to tell me, “You have a lucky nose”, because its particular shape was a lucky one, according to Chinese lore.’ Growing up in the American Midwest, it dawned on Young that the fleeting luck that Americans often talked about – a luck that seemed to visit the same person at certain times (‘I got lucky on that test!’) but not others (‘I got caught in traffic before my interview!’) – was not equivalent to the unchanging, stable luck her mother saw in her daughter, her nose being an advertisement of its existence within.

‘It’s something that I have that’s a possession of mine, that can be more relied upon than just dumb luck,’ says Young. The distinction stuck with her. You might think someone with a lucky nose wouldn’t roll up their sleeves to work hard – why bother? – but here’s another cultural difference in perceptions of luck. ‘In Chinese culture,’ she says, ‘hard work can go hand-in-hand with being lucky. The belief system accommodates both.’

On the other hand, because Westerners see effort and good fortune as taking up opposite corners of the ring, they are ambivalent about luck. They might pray for it and sincerely wish others they care about ‘Good luck!’ but sometimes they just don’t want to think of themselves as lucky. They’d rather be deserving. The fact that they live in a society that is neither random nor wholly meritocratic makes for an even messier slamdance between ‘hard work’ and ‘luck’. Case in point: when a friend gets into a top law or medical school, we might say: ‘Congratulations! You’ve persevered. You deserve it.’ Were she not to get in, we would say: ‘Acceptance is arbitrary. Everyone’s qualified these days – it’s the luck of the draw.’

Read the entire article here.

Image: Four-leaf clover. Some consider it a sign of god luck. Courtesy of Phyzome.

BigBang

Non-Adaptive Evolution of the Very Small

Is every feature that arises from evolution an adaptation?  Some evolutionary biologists think not. That is, some traits arising from the process of natural section may be due to random occurrences that natural selection failed to discard. And, it seems that smaller organisms show this quite well. To many adaptationists this is heretical — but too some researchers it opens a new, fruitful avenue of inquiry, and may lead to a fine tuning in our understanding of the evolutionary process.

From New Scientist:

I have spent my life working on slime moulds and they sent me a message that started me thinking. What puzzled me was that two different forms are found side-by-side in the soil everywhere from the tundra to the tropics. The obvious difference lies in the tiny stalks that disperse their spores. In one species this fruiting body is branched, in the other it is not.

I had assumed that the branched and the unbranched forms occupied separate ecological niches but I could not imagine what those niches might be. Perhaps there were none and neither shape had an advantage over the other, as far as natural selection was concerned.

I wrote this up and sent it to a wise and respected friend who responded with a furious letter saying that my conclusion was absurd: it was easy to imagine ways in which the two kinds of stalks might be separate adaptations and co-exist everywhere in the soil. This set me thinking again and I soon realised that both my position and his were guesses. They were hypotheses and neither could be proved.

There is no concept that is more central to evolution than natural selection, so adding this extra dimension of randomness was heresy. Because of the overwhelming success of Darwin’s natural selection, biologists – certainly all evolutionary biologists – find it hard to believe that a feature of any organism can have arisen (with minor exceptions) in any other way. Natural selection favours random genetic mutations that offer an advantage, therefore many people believe that all properties of an organism are an adaptation. If one cannot find the adaptive reason for a feature of an organism, one should just assume that there was once one, or that there is one that will be revealed in the future.

This matter has created some heated arguments. For example, the renowned biologists Stephen Jay Gould and Richard Lewontin wrote an inflammatory paper in 1979 attacking adaptionists for being like Dr Pangloss, the incurable optimist in Voltaire’s 1759 satire Candide. While their point was well taken, its aggressive tone produced counterattacks. Adaptionists assume that every feature of an organism arises as an adaption, but I assume that some features are the results of random mutations that escape being culled by natural selection. This is what I was suggesting for the branched and unbranched fruiting bodies of the slime moulds.

How can these organisms escape the stranglehold of selection? One explanation grabbed me and I have clung to it ever since; in fact it is the backbone of my new book. The reason that these organisms might have shapes that are not governed by natural selection is because they are so small. It turns out there are good reasons why this might be the case.

Development is a long, slow process for large organisms. Humans spend nine months in utero and keep growing in different ways for a long time after birth. An elephant’s gestation is even longer (about two years) and a mouse’s much shorter, but they are all are vastly longer than a single-cell microorganism. Such small forms may divide every few hours; at most their development may span days, but whatever it is it will be a small fraction of that of a larger, more complex organism.

Large organisms develop in a series of steps usually beginning with the fertilisation of an egg that then goes through many cell divisions and an increase in size of the embryo, with many twists and turns as it progresses towards adulthood. These multitudinous steps involve the laying down of complex organs such as a heart or an eye.

Building a complex organism is an immense enterprise, and the steps are often interlocked in a sequence so that if an earlier step fails through a deleterious mutation, the result is very simple: the death of the embryo. I first came across this idea in a 1965 book by Lancelot Law Whyte called Internal Factors in Evolution and have been mystified ever since why the idea has been swallowed by oblivion. His thesis was straightforward. Not only is there selection of organisms in the environment – Darwinian natural selection, which is external – but there is also continuous internal selection during development. Maybe the idea was too simple and straightforward to have taken root.

This fits in neatly with my contention that the shape of microorganisms is more affected by randomness than for large, complex organisms. Being small means very few development steps, with little or no internal selection. The effect of a mutation is likely to be immediately evident in the external morphology, so adult variants are produced with large numbers of different shapes and there is an increased chance that some of these will be untouched by natural selection.

Compare this with what happens in a big, complex organism – a mammal, say. Only those mutations that occur at a late stage of development are likely to be viable – eye or hair colour in humans are obvious examples. Any unfavourable mutation that occurs earlier in development will likely be eliminated by internal selection.

Let us now examine the situation for microorganisms. What is the evidence that their shapes are less likely to be culled by natural selection? The best examples come from organisms that make mineral shells: Radiolaria (pictured) and diatoms with their silica skeletons and Foraminifera with their calciferous shells. About 50,000 species of radiolarians have been described, 100,000 species of diatoms and some 270,000 species among the Foraminifera – all with vastly different shapes. For example, radiolarian skeletons can be shaped like spiny balls, bells, crosses and octagonal pyramids, to name but a few.

If you are a strict adaptionist, you have to find a separate explanation for each shape. If you favour my suggestion that their shapes arose through random mutation and there is little or no selection, the problem vanishes. It turns out that this very problem concerned Darwin. In the third (and subsequent) editions of On the Origin of Species he has a passage that almost takes the wind out of my sails:

“If it were no advantage, these forms would be left by natural selection unimproved or but little improved; and might remain for indefinite ages in their present little advanced condition. And geology tells us that some of the lowest forms, as the infusoria and rhizopods, have remained for an enormous period in nearly their present state.”

Read the entire article here.

Idea Soup

Past Experience is Good; Random Decision-Making is Better

We all know that making decisions from past experience is wise. We learn from the benefit of hindsight. We learn to make small improvements or radical shifts in our thinking and behaviors based on history and previous empirical evidence. Stock market gurus and investment mavens will tell you time after time that they have a proven method — based on empirical evidence and a lengthy, illustrious track record — for picking the next great stock or investing your hard-earned retirement funds.

Yet, empirical evidence shows that chimpanzees throwing darts at the WSJ stock pages are just as good at stock market tips as we humans (and the “masters of the universe”). So, it seems that random decision-making can be just as good, if not better, than wisdom and experience.

From the Guardian:

No matter how much time you spend reading the recent crop of books on How To Decide or How To Think Clearly, you’re unlikely to encounter glowing references to a decision-making system formerly used by the Azande of central Africa. Faced with a dilemma, tribespeople would force poison down the neck of a chicken while asking questions of the “poison oracle”; the chicken answered by surviving (“yes”) or expiring (“no”). Clearly, this was cruel to chickens. That aside, was it such a terrible way to choose among options? The anthropologist EE Evans-Pritchard, who lived with the Azande in the 1920s, didn’t think so. “I always kept a supply of poison [and] we regulated our affairs in accordance with the oracle’s decisions,” he wrote, adding drily: “I found this as satisfactory a way of running my home and affairs as any other I know of.” You could dismiss that as a joke. After all, chicken-poisoning is plainly superstition, delivering random results. But what if random results are sometimes exactly what you need?

The other day, US neuroscientists published details of experiments on rats, showing that in certain unpredictable situations, they stop trying to make decisions based on past experience. Instead, a circuit in their brains switches to “random mode”. The researchers’ hunch is that this serves a purpose: past experience is usually helpful, but when uncertainty levels are high, it can mislead, so randomness is in the rats’ best interests. When we’re faced with the unfamiliar, experience can mislead humans, too, partly because we filter it through various irrational biases. According to those books on thinking clearly, we should strive to overcome these biases, thus making more rational calculations. But there’s another way to bypass our biased brains: copy the rats, and choose randomly.

In certain walks of life, the usefulness of randomness is old news: the stock market, say, is so unpredictable that, to quote the economist Burton Malkiel, “a blindfolded monkey throwing darts at a newspaper’s financial pages could select a portfolio that would do as well as one carefully selected by experts”. (This has been tried, with simulated monkeys, andthey beat the market.) But, generally, as Michael Schulson put it recentlyin an Aeon magazine essay, “We take it for granted that the best decisions stem from empirical analysis and informed choice.” Yet consider, he suggests, the ancient Greek tradition of filling some government positions by lottery. Randomness disinfects a process that might be dirtied by corruption.

Randomness can be similarly useful in everyday life. For tiny choices, it’s a time-saver: pick randomly from a menu, and you can get back to chatting with friends. For bigger ones, it’s an acknowledgment of how little one can ever know about the complex implications of a decision. Let’s be realistic: for the biggest decisions, such as whom to marry, trusting to randomness feels absurd. But if you can up the randomness quotient for marginally less weighty choices, especially when uncertainty prevails, you may find it pays off. Though kindly refrain from poisoning any chickens.

Read the entire article here.

Idea Soup, tD

Measuring Antifragility

Nassim Nicholas Taleb, one of our favorite thinkers and writers over here at theDiagonal recently published Antifragile, his follow-up to his successful “black swan” title Black Swan. In Antifragile Taleb argues that some things thrive when subjected to volatility, disorder and uncertainty. He labels the positive reaction to these external stressors, antifragility. (Ironically, this book was published by Random House).

In his essay, excerpted below, Taleb summarizes the basic tenets of antifragility and the payoff that we would gain from its empirical measurement. This would certainly represent a leap forward, from our persistent and misguided focus on luck in research, relationships and business.

[div class=attrib]From Edge.org:[end-div]

Something central, very central, is missing in historical accounts of scientific and technological discovery. The discourse and controversies focus on the role of luck as opposed to teleological programs (from telos, “aim”), that is, ones that rely on pre-set direction from formal science. This is a faux-debate: luck cannot lead to formal research policies; one cannot systematize, formalize, and program randomness. The driver is neither luck nor direction, but must be in the asymmetry (or convexity) of payoffs, a simple mathematical property that has lied hidden from the discourse, and the understanding of which can lead to precise research principles and protocols.

MISSING THE ASYMMETRY

The luck versus knowledge story is as follows. Ironically, we have vastly more evidence for results linked to luck than to those coming from the teleological, outside physics—even after discounting for the sensationalism. In some opaque and nonlinear fields, like medicine or engineering, the teleological exceptions are in the minority, such as a small number of designer drugs. This makes us live in the contradiction that we largely got here to where we are thanks to undirected chance, but we build research programs going forward based on direction and narratives. And, what is worse, we are fully conscious of the inconsistency.

The point we will be making here is that logically, neither trial and error nor “chance” and serendipity can be behind the gains in technology and empirical science attributed to them. By definition chance cannot lead to long term gains (it would no longer be chance); trial and error cannot be unconditionally effective: errors cause planes to crash, buildings to collapse, and knowledge to regress.

The beneficial properties have to reside in the type of exposure, that is, the payoff function and not in the “luck” part: there needs to be a significant asymmetry between the gains (as they need to be large) and the errors (small or harmless), and it is from such asymmetry that luck and trial and error can produce results. The general mathematical property of this asymmetry is convexity (which is explained in Figure 1); functions with larger gains than losses are nonlinear-convex and resemble financial options. Critically, convex payoffs benefit from uncertainty and disorder. The nonlinear properties of the payoff function, that is, convexity, allow us to formulate rational and rigorous research policies, and ones that allow the harvesting of randomness.

OPAQUE SYSTEMS AND OPTIONALITY

Further, it is in complex systems, ones in which we have little visibility of the chains of cause-consequences, that tinkering, bricolage, or similar variations of trial and error have been shown to vastly outperform the teleological—it is nature’s modus operandi. But tinkering needs to be convex; it is imperative. Take the most opaque of all, cooking, which relies entirely on the heuristics of trial and error, as it has not been possible for us to design a dish directly from chemical equations or reverse-engineer a taste from nutritional labels. We take hummus, add an ingredient, say a spice, taste to see if there is an improvement from the complex interaction, and retain if we like the addition or discard the rest. Critically we have the option, not the obligation to keep the result, which allows us to retain the upper bound and be unaffected by adverse outcomes.

This “optionality” is what is behind the convexity of research outcomes. An option allows its user to get more upside than downside as he can select among the results what fits him and forget about the rest (he has the option, not the obligation). Hence our understanding of optionality can be extended to research programs — this discussion is motivated by the fact that the author spent most of his adult life as an option trader. If we translate François Jacob’s idea into these terms, evolution is a convex function of stressors and errors —genetic mutations come at no cost and are retained only if they are an improvement. So are the ancestral heuristics and rules of thumbs embedded in society; formed like recipes by continuously taking the upper-bound of “what works”. But unlike nature where choices are made in an automatic way via survival, human optionality requires the exercise of rational choice to ratchet up to something better than what precedes it —and, alas, humans have mental biases and cultural hindrances that nature doesn’t have. Optionality frees us from the straightjacket of direction, predictions, plans, and narratives. (To use a metaphor from information theory, if you are going to a vacation resort offering you more options, you can predict your activities by asking a smaller number of questions ahead of time.)

While getting a better recipe for hummus will not change the world, some results offer abnormally large benefits from discovery; consider penicillin or chemotherapy or potential clean technologies and similar high impact events (“Black Swans”). The discovery of the first antimicrobial drugs came at the heel of hundreds of systematic (convex) trials in the 1920s by such people as Domagk whose research program consisted in trying out dyes without much understanding of the biological process behind the results. And unlike an explicit financial option for which the buyer pays a fee to a seller, hence tend to trade in a way to prevent undue profits, benefits from research are not zero-sum.

[div class=attrib]Read the entire article following the jump.[end-div]

[div class=attrib]Image: Antifragile by Naseem Nicholas Taleb, book cover. Courtesy of the author / Random House / Barnes & Noble.[end-div]

Idea Soup

Why Do Some Videos Go Viral, and Others Not?

Some online videos and stories are seen by tens or hundreds of millions, yet others never see the light of day. Advertisers and reality star wannabes search daily for the secret sauce that determines the huge success of one internet meme over many others. However, much to the frustration of the many agents to the “next big thing”, several fascinating new studies point at nothing more than simple randomness.

[div class=attrib]From the New Scientist:[end-div]

WHAT causes some photos, videos, and Twitter posts to spread across the internet like wildfire while others fall by the wayside? The answer may have little to do with the quality of the information. What goes viral may be completely arbitrary, according to a controversial new study of online social networks.

By analysing 120 million retweets – repostings of users’ messages on Twitter – by 12.5 million users of the social network, researchers at Indiana University, Bloomington, learned the mechanisms by which memes compete for user interest, and how information spreads.

Using this insight, the team built a computer simulation designed to mimic Twitter. In the simulation, each tweet or message was assigned the same value and retweets were performed at random. Despite this, some tweets became incredibly popular and were persistently reposted, while others were quickly forgotten.

The reason for this, says team member Filippo Menczer, is that the simulated users had a limited attention span and could only view a portion of the total number of tweets – as is the case in the real world. Tweets selected for retweeting would be more likely to be seen by a user and re-posted. After a few iterations, a tweet becomes significantly more prevalent than those not retweeted. Many users see the message and retweet it further.

“When a meme starts to get popular it displaces other memes; you start to pay attention to the popular meme and don’t pay attention to other things because you have only so much attention,” Menczer says. “It’s similar to when a big news story breaks, you don’t hear about other things that happened on that day.”

Katherine Milkman of the University of Pennsylvania in Philadelphia disagrees. “[Menczer’s study] says that all of the things that catch on could be truly random but it doesn’t say they have to be,” says Milkman, who co-authored a paper last year examining how emotions affect meme sharing.

Milkman’s study analysed 7000 articles that appeared in the New York Times over a three-month period. It found that articles that aroused readers’ emotions were more likely to end up on the website’s “most emailed” list. “Anything that gets you fired up, whether positive or negative, will lead you to share it more,” Milkman says.

[div class=attrib]Read the entire article after the jump.[end-div]

[div class=attrib]Image: Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets is a book by Nassim Nicholas Taleb. Courtesy of Wikipedia.[end-div]

Book Reviews

Book Review: The Drunkard’s Walk: How Randomness Rules Our Lives. Leonard Mlodinow

Leonard Mlodinow weaves a compelling path through the world of statistical probability showing us how the laws of chance affect our lives on personal and grande scales. Mlodinow skillfully illustrates randomness and its profound implications by presenting complex mathematical constructs in language for the rest of us (non-mathematicians), without dumbing-down this important subject.

The book defines many of the important mathematical concepts behind randomness and exposes the key fallacies that often blind us as we wander through life on our “drunkard’s walk”. The law of large numbers, the prosecutor’s fallacy, conditional probability, the availability bias and bell curves were never so approachable.

Whether it’s a deluded gambler, baseball star on a “winning streak” or a fortunate CEO wallowing in the good times, Mlodinow debunks the common conceptions that skill, planning and foresight result in any significant results beyond pure chance. With the skill of a storyteller Mlodinow shows us how polls, grades, ratings and even measures of corporate success are far less objective and reliable than we ought to believe. Lords of Wall Street take notice, the secrets of your successes are not all that they seem.