She scolded me for not being observant enough, for missing out on all the cues and subtleties that have been flying around within our social group. In return, I gave her my best nonchalant raised eyebrow look and told her that there is nothing wrong with my selective horse-blinders attitude to other people's relationships around me. The psyche of a women. Her peripheral vision and hearing.
16.9.12
29.7.12
RI 189TH FOUNDER’S DAY Speech
[quote]
SPEECH BY PS (FOREIGN AFFAIRS) BILAHARI KAUSIKAN AT RAFFLES INSTITUTION’S 189TH FOUNDER’S DAY ON 21 JULY 2012 (SATURDAY) AT 9 AM AT ALBERT HONG HALL, RAFFLES INSTITUTION
1 When your Principal, in a reckless act of folly, asked me to be Guest-of-Honour at this 189th Founder's Day, my first instinct was to do us both a favour and refuse. But I hesitated and in an instant was lost. The temptation to savour the irony was too great. For what I am about to say, I absolve her of all responsibility.
2 My comrades and I spent our six years in Raffles Institution waging insurgency against all established authority. At a very tender age one of our teachers told us we were all born to be hanged. And if that extreme did not come to pass -- perhaps I should say, has not yet come to pass -- several of us were at least caned. Our then Principal failed to achieve his dearest ambition of getting us all expelled only due to our dumb luck.
3 So here I stand before you, living testimony to the role of chance and serendipity in life; a role more often than not, insufficiently acknowledged if not ignored, particularly by Singaporeans of a certain ilk. And that is my theme.
4 Eighty-five years ago an American writer by the name of Thornton Wilder published a short novel entitled The Bridge of San Luis Rey. The book has never been out of print, but deserves to be better known.
5 The novel begins at noon on a certain day in 1714 when a bridge in Peru -- "the finest bridge in all Peru", writes Wilder -- inexplicably collapses and five people who happen at that moment to be crossing, plummet to their deaths.
6 The tragedy is witnessed by a devout Franciscan monk, in Peru for missionary work among the natives, who immediately asks himself “Why did this happen to those five?”
7 The monk is convinced that it was not a random event but some manifestation of God's Will for some greater end and vows to investigate to so as to prove to the natives the necessity of divine purpose. But his investigation runs afoul of the Inquisition and he is burnt at the stake.
8 Wilder poses, but never directly answers, the question: “Is there a direction and meaning in lives beyond the individual's own will?” The point, of course, is that it could have been anyone of us on that metaphorical bridge.
9 I do not think that there is any particular meaning, pattern or direction, divine or secular, in the drift of human events. History, as Winston Churchill is reported to have remarked, is just one damned thing after another. The innocent die young and the wicked flourish; and not necessarily in equal measure either because to the wicked the innocent are often prey.
10 The world is far too complex a place to be comprehended in any holistic way by the human mind. It is made up of too many moving parts interacting in too many unpredictable ways for human reason to grasp.
11 I mean, of course, the social world: the world of human interactions, human relationships and human institutions; of love and hatred, politics and economics, war and peace, infused with emotions like anger, pity, joy and sorrow, and not the material world of rocks and stones and trees and the earth's diurnal course.
12 In the material world, the apple will always fall whether or not Newton was there to observe it. In the material world, all phenomena must ultimately conform to the laws of physics. In the material world, when we return to earth and ashes, we too will confirm to the laws of physics.
13 But in the meantime we inhabit a social world of sentient beings who observe, think and respond so that our every effort to act or comprehend alters what we try to comprehend and every thought and action begets a never ending, ever shifting kaleidoscope of unpredictable possibilities that makes all social science an oxymoron.
14 Reason may distinguish man from beast, but the sum of the interactions of different reasons; of many logics, is only coincidentally and occasionally logical. That is why actions always have unintended consequences even if they are not always immediately apparent, and our best laid plans and most fervent hopes are constantly ambushed by chance and events.
15 Most things eventually fail. The shade of Ozymandias hovers unseen but omnipresent over every human enterprise, biding its time.
16 The ancient Greeks advised us to call no man happy until he was dead. This is good advice. We can be reasonably certain of something only after it has occurred. The only true knowledge is historical, and even then there is always room for argument over interpretation. None of us ever sees or understands the same thing, no matter how conscientiously we try to observe or communicate.
17 As I stand here speaking to you, at least three different things are occurring simultaneously: first, what I think; second, what I say to convey what I think which, whether because of the limitations of language or by design, will not always be the same as what I think: deception and self-deception are intrinsic parts of human nature; and third what you hear and understand of what I had intended to convey which is again not necessarily the same thing.
18 One could call this, after the title of a short story by the Japanese author Ryunosuke Akutagawa, the Rashomon phenomenon. It makes for a world without fixed meaning, which accentuates its fundamental incomprehensibility. A world in which the past can only be partially known, the present is largely unknown and the future certainly unknowable.
19 None of us asked to be born. Yet having had life thrust upon us, we must, unless bent on suicide, nevertheless live. Although we can only, if dimly and darkly, know backwards, we have to live forwards.
20 No one can live in a constant Hamlet-like state of existential doubt. We must profess a certainty that we do not necessarily feel. To keep the metaphysical horror of unfathomable meaninglessness at bay, we all, singly or collectively, consciously or unconsciously, adopt mental frameworks to simplify a complex reality in order to deal with it.
21 Since the Enlightenment of the 17th Century, belief in Reason has replaced belief in God as the primary organizing mental framework of society. We are all the creatures of this western defined modernity and the most successful of the non-western countries, Singapore among them, are precisely those who have embraced it the most closely.
22 Reason's children include law and justice, philosophy, literature and the arts, economics and other social sciences and even the very belief in reason, progress, technology and science. But the fundamental mode of thought that underpins these trappings of reason is still theological in that whether our belief is in Reason or in God, it is still mere belief and not epistemologically provable beyond all doubt. There is no end to philosophy any more than there can be an end to history.
23 Stated in another way, none of Reason's children have an autonomous reality separate from our apprehensions of them. They are socially constructed artefacts; frameworks of ideas that we have chosen to believe in, in order to comprehend the world and comprehend in order live in a particular way.
24 Their utility is thus purely instrumental. They are at best all only partially and contingently right which means, of course, that they are all also always at least partially wrong. That includes, by the way, the ideas I am presently expounding.
25 I advance these arguments not to instil cynicism or despair but to suggest the possibility of liberation and hope.
26 A rock is forever only a rock. But human beings are defined by their potentialities, and since there is no predetermined meaning to the unfolding of events, the potentialities are equally boundless. Were it not so, Singapore should not exist as a sovereign and independent country.
27 The only meaning in life that can exist is that which we create for ourselves. And unless we want our lives to be merely a slow, selfish dying, we ought to try to create some meaning larger than ourselves.
28 This is, to my mind, an absolute duty imposed by the human condition, even if we know that uncertainty and error are constants and that we are always writing on sand before the advancing tide. Our duties to our families, our friends and our country endure when even hope is dead.
29 I am sure that by now many of you are harbouring a thought that you are too well brought up to speak out loud: this idiot exaggerates.
30 Of course, I exaggerate. But only a little, and only for clarity's sake and not to distort or mislead. So let me restate my essential point in a different way.
31 Do not confuse the depth of sincerity with which you or others hold an idea, or the number of people who sincerely hold an idea, with its validity. Sincerity is an over-rated virtue, if indeed it is a virtue.
32 All of you may be suddenly seized with the sincere conviction that that pigs should fly. But pigs will nevertheless never sprout wings no matter how devoutly you hope for them to escape the surly bonds of earth.
33 And if you, ignoring the possibility of error, sincerely believe that pigs ought to fly; or that God's Will has been revealed to you; or that you are one of the elect to whom the direction of History's cunning passages has been vouchsafed, then it is but a tiny step to being convinced that anyone who does not share your conviction is not just ignorant but evil. Then for the greater glory of PIGS or HISTORY or GOD, all spelt in capital letters, it is only a tinier further step to seeing it as your bounden DUTY, again spelt with capitals, to expunge the evil.
34 And it all inevitably ends as Wilder's poor monk did, in flames at the stake.
35 Rather than sincerity, if we want to do some trifling and ephemeral good or at least to minimize harm, we should approach life with an ironic and humane scepticism.
36 Irony to ensure that we retain a sense of proportion and as ballast against the inevitability of unintended consequences: today's error being the correction of yesterday's error. Humanity so that we may empathize with logics other than our own, if only to better manoeuvre to impose our will because in a world of competing logics, if we hope to do any good, we cannot hope to do so by logic alone. And scepticism because the possibility of deception, our own self-deceptions if not those of others, casts constant shadows over every human action.
37 I have chosen to dwell on this at what you may consider inordinate length, because Raffles Institution likes to consider itself unique. Ladies and Gentlemen, I am sorry to inform you that RI is no longer unique.
38 You are now only one of a number of similar elite educational institutions from which will come a disproportionate number of scholarship recipients and a disproportionate number of leaders in the civil service, the professions, business, the Arts and the academy. And all these institutions are united by a certain sense of entitlement, possibly so profound as to be quite unconscious.
39 I do not blame you for this. All of you are highly intelligent. You will be very well educated. And the odds are that you will be more than averagely successful in your careers.
40 But understand that you will therefore also be more vulnerable to the curse of the highly intelligent, highly educated and highly successful: this curse is the illusion of certainty; the conviction of the omnipotence of your ideas.
41 This is the delusion that your ideas or words are validated by mere virtue by being thought or uttered by you! YOU and not some lesser being. And the more intelligent and the more successful and the more highly educated, the deeper the delusion. "The learned", Adam Smith is reported to have said, “ignore the evidence of their senses to preserve the coherence of the ideas of their imagination.”
42 Shortly after the 2008-2009 financial crisis, Alan Greenspan, the former Chairman of the US Federal Reserve, a powerful and erudite man, confessed in testimony to a Senate. Hearing that his intellectual assumptions of a lifetime had been shaken and he was still trying to understand what happened. I do not know if he has since come to any conclusions. But it was clear that prior to the near global disaster, he had never even faintly contemplated the possibility that his beliefs may have been in error. We are all still paying the price for his certainties.
43 Yours will be a generation that that will live through times of more than usual uncertainty.
44 A global transition of power and ideas is underway. Transition to what, no one can yet say. We have no maps and will have to improvise our way forward the best we can. It will be a transition measured in decades and not just a few years, and it is your misfortune that it is occurring as the technology of the internet is making us solipsistic.
45 The internet conflates and confuses our opinion with information and tempts us to immerse ourselves only in a circle of those who share and reinforce our own interests and views. It shortens attention spans and privileges the new and novel over any notion of lasting value. Social media like Facebook have perverted the common meaning of ‘friend’ and ‘like’ beyond all recognition. Only a solipsist or, what is much the same thing, a narcissist, would think that what he or she had for lunch would be of wider interest; and only those with vacuous minds would be interested. And this at a time when the safe navigation of uncharted waters requires a prudent modesty, openness and some minimal capacity for sustained thought.
46 And yet the internet and its associated technologies is indispensible to modern life. We need it to prosper. But what its ultimate effects will be on society, on governance, on international relations, on the very way we think, no one yet knows.
47 I certainly have no answers. As you, the anointed ones, ready yourselves to assume authority and responsibility under these challenging circumstances. I can do no more than to remind you of what Sir Olivier Cromwell wrote to the Synod of the Church of Scotland in 1650. He was trying to persuade the Scots not to embrace the Royalist cause of King Charles the Second and so avert civil war.
48 Gentlemen, he wrote, “I beseech you, in the bowels of Christ” -- and I should explain that in the 17th Century the bowels were considered to be the seat of pity or the gentler emotions -- Gentlemen, Cromwell wrote, “I beseech you, in the bowels of Christ, think it possible that you may be mistaken”.
49 So, Ladies and Gentlemen of the 21st Century, I too beseech you from whatever portion of anatomy you consider most dear, think it possible that you may be mistaken.
50 Before I conclude, you may wish to know how it all ended.
51 Cromwell's advice was not heeded. Shortly thereafter, the third English Civil War broke out. This set in motion a historical trajectory of political, social and economic changes that led to modern Britain, the industrial revolution, the East India Company, Sir Stamford Raffles, the British Empire, the founding of Singapore and ultimately, you and I.
52 And all because good advice fell on deaf ears.
53 What better way to appreciate the irony and contingency of events than to ponder what may have happened if Cromwell's advice was in fact taken and civil war avoided. And as you do so, consider also the possibility that you may be mistaken when you think you are mistaken.
54 And with that final paradox I will end.
55 Thank you for listening to me
wow.
SPEECH BY PS (FOREIGN AFFAIRS) BILAHARI KAUSIKAN AT RAFFLES INSTITUTION’S 189TH FOUNDER’S DAY ON 21 JULY 2012 (SATURDAY) AT 9 AM AT ALBERT HONG HALL, RAFFLES INSTITUTION
1 When your Principal, in a reckless act of folly, asked me to be Guest-of-Honour at this 189th Founder's Day, my first instinct was to do us both a favour and refuse. But I hesitated and in an instant was lost. The temptation to savour the irony was too great. For what I am about to say, I absolve her of all responsibility.
2 My comrades and I spent our six years in Raffles Institution waging insurgency against all established authority. At a very tender age one of our teachers told us we were all born to be hanged. And if that extreme did not come to pass -- perhaps I should say, has not yet come to pass -- several of us were at least caned. Our then Principal failed to achieve his dearest ambition of getting us all expelled only due to our dumb luck.
3 So here I stand before you, living testimony to the role of chance and serendipity in life; a role more often than not, insufficiently acknowledged if not ignored, particularly by Singaporeans of a certain ilk. And that is my theme.
4 Eighty-five years ago an American writer by the name of Thornton Wilder published a short novel entitled The Bridge of San Luis Rey. The book has never been out of print, but deserves to be better known.
5 The novel begins at noon on a certain day in 1714 when a bridge in Peru -- "the finest bridge in all Peru", writes Wilder -- inexplicably collapses and five people who happen at that moment to be crossing, plummet to their deaths.
6 The tragedy is witnessed by a devout Franciscan monk, in Peru for missionary work among the natives, who immediately asks himself “Why did this happen to those five?”
7 The monk is convinced that it was not a random event but some manifestation of God's Will for some greater end and vows to investigate to so as to prove to the natives the necessity of divine purpose. But his investigation runs afoul of the Inquisition and he is burnt at the stake.
8 Wilder poses, but never directly answers, the question: “Is there a direction and meaning in lives beyond the individual's own will?” The point, of course, is that it could have been anyone of us on that metaphorical bridge.
9 I do not think that there is any particular meaning, pattern or direction, divine or secular, in the drift of human events. History, as Winston Churchill is reported to have remarked, is just one damned thing after another. The innocent die young and the wicked flourish; and not necessarily in equal measure either because to the wicked the innocent are often prey.
10 The world is far too complex a place to be comprehended in any holistic way by the human mind. It is made up of too many moving parts interacting in too many unpredictable ways for human reason to grasp.
11 I mean, of course, the social world: the world of human interactions, human relationships and human institutions; of love and hatred, politics and economics, war and peace, infused with emotions like anger, pity, joy and sorrow, and not the material world of rocks and stones and trees and the earth's diurnal course.
12 In the material world, the apple will always fall whether or not Newton was there to observe it. In the material world, all phenomena must ultimately conform to the laws of physics. In the material world, when we return to earth and ashes, we too will confirm to the laws of physics.
13 But in the meantime we inhabit a social world of sentient beings who observe, think and respond so that our every effort to act or comprehend alters what we try to comprehend and every thought and action begets a never ending, ever shifting kaleidoscope of unpredictable possibilities that makes all social science an oxymoron.
14 Reason may distinguish man from beast, but the sum of the interactions of different reasons; of many logics, is only coincidentally and occasionally logical. That is why actions always have unintended consequences even if they are not always immediately apparent, and our best laid plans and most fervent hopes are constantly ambushed by chance and events.
15 Most things eventually fail. The shade of Ozymandias hovers unseen but omnipresent over every human enterprise, biding its time.
16 The ancient Greeks advised us to call no man happy until he was dead. This is good advice. We can be reasonably certain of something only after it has occurred. The only true knowledge is historical, and even then there is always room for argument over interpretation. None of us ever sees or understands the same thing, no matter how conscientiously we try to observe or communicate.
17 As I stand here speaking to you, at least three different things are occurring simultaneously: first, what I think; second, what I say to convey what I think which, whether because of the limitations of language or by design, will not always be the same as what I think: deception and self-deception are intrinsic parts of human nature; and third what you hear and understand of what I had intended to convey which is again not necessarily the same thing.
18 One could call this, after the title of a short story by the Japanese author Ryunosuke Akutagawa, the Rashomon phenomenon. It makes for a world without fixed meaning, which accentuates its fundamental incomprehensibility. A world in which the past can only be partially known, the present is largely unknown and the future certainly unknowable.
19 None of us asked to be born. Yet having had life thrust upon us, we must, unless bent on suicide, nevertheless live. Although we can only, if dimly and darkly, know backwards, we have to live forwards.
20 No one can live in a constant Hamlet-like state of existential doubt. We must profess a certainty that we do not necessarily feel. To keep the metaphysical horror of unfathomable meaninglessness at bay, we all, singly or collectively, consciously or unconsciously, adopt mental frameworks to simplify a complex reality in order to deal with it.
21 Since the Enlightenment of the 17th Century, belief in Reason has replaced belief in God as the primary organizing mental framework of society. We are all the creatures of this western defined modernity and the most successful of the non-western countries, Singapore among them, are precisely those who have embraced it the most closely.
22 Reason's children include law and justice, philosophy, literature and the arts, economics and other social sciences and even the very belief in reason, progress, technology and science. But the fundamental mode of thought that underpins these trappings of reason is still theological in that whether our belief is in Reason or in God, it is still mere belief and not epistemologically provable beyond all doubt. There is no end to philosophy any more than there can be an end to history.
23 Stated in another way, none of Reason's children have an autonomous reality separate from our apprehensions of them. They are socially constructed artefacts; frameworks of ideas that we have chosen to believe in, in order to comprehend the world and comprehend in order live in a particular way.
24 Their utility is thus purely instrumental. They are at best all only partially and contingently right which means, of course, that they are all also always at least partially wrong. That includes, by the way, the ideas I am presently expounding.
25 I advance these arguments not to instil cynicism or despair but to suggest the possibility of liberation and hope.
26 A rock is forever only a rock. But human beings are defined by their potentialities, and since there is no predetermined meaning to the unfolding of events, the potentialities are equally boundless. Were it not so, Singapore should not exist as a sovereign and independent country.
27 The only meaning in life that can exist is that which we create for ourselves. And unless we want our lives to be merely a slow, selfish dying, we ought to try to create some meaning larger than ourselves.
28 This is, to my mind, an absolute duty imposed by the human condition, even if we know that uncertainty and error are constants and that we are always writing on sand before the advancing tide. Our duties to our families, our friends and our country endure when even hope is dead.
29 I am sure that by now many of you are harbouring a thought that you are too well brought up to speak out loud: this idiot exaggerates.
30 Of course, I exaggerate. But only a little, and only for clarity's sake and not to distort or mislead. So let me restate my essential point in a different way.
31 Do not confuse the depth of sincerity with which you or others hold an idea, or the number of people who sincerely hold an idea, with its validity. Sincerity is an over-rated virtue, if indeed it is a virtue.
32 All of you may be suddenly seized with the sincere conviction that that pigs should fly. But pigs will nevertheless never sprout wings no matter how devoutly you hope for them to escape the surly bonds of earth.
33 And if you, ignoring the possibility of error, sincerely believe that pigs ought to fly; or that God's Will has been revealed to you; or that you are one of the elect to whom the direction of History's cunning passages has been vouchsafed, then it is but a tiny step to being convinced that anyone who does not share your conviction is not just ignorant but evil. Then for the greater glory of PIGS or HISTORY or GOD, all spelt in capital letters, it is only a tinier further step to seeing it as your bounden DUTY, again spelt with capitals, to expunge the evil.
34 And it all inevitably ends as Wilder's poor monk did, in flames at the stake.
35 Rather than sincerity, if we want to do some trifling and ephemeral good or at least to minimize harm, we should approach life with an ironic and humane scepticism.
36 Irony to ensure that we retain a sense of proportion and as ballast against the inevitability of unintended consequences: today's error being the correction of yesterday's error. Humanity so that we may empathize with logics other than our own, if only to better manoeuvre to impose our will because in a world of competing logics, if we hope to do any good, we cannot hope to do so by logic alone. And scepticism because the possibility of deception, our own self-deceptions if not those of others, casts constant shadows over every human action.
37 I have chosen to dwell on this at what you may consider inordinate length, because Raffles Institution likes to consider itself unique. Ladies and Gentlemen, I am sorry to inform you that RI is no longer unique.
38 You are now only one of a number of similar elite educational institutions from which will come a disproportionate number of scholarship recipients and a disproportionate number of leaders in the civil service, the professions, business, the Arts and the academy. And all these institutions are united by a certain sense of entitlement, possibly so profound as to be quite unconscious.
39 I do not blame you for this. All of you are highly intelligent. You will be very well educated. And the odds are that you will be more than averagely successful in your careers.
40 But understand that you will therefore also be more vulnerable to the curse of the highly intelligent, highly educated and highly successful: this curse is the illusion of certainty; the conviction of the omnipotence of your ideas.
41 This is the delusion that your ideas or words are validated by mere virtue by being thought or uttered by you! YOU and not some lesser being. And the more intelligent and the more successful and the more highly educated, the deeper the delusion. "The learned", Adam Smith is reported to have said, “ignore the evidence of their senses to preserve the coherence of the ideas of their imagination.”
42 Shortly after the 2008-2009 financial crisis, Alan Greenspan, the former Chairman of the US Federal Reserve, a powerful and erudite man, confessed in testimony to a Senate. Hearing that his intellectual assumptions of a lifetime had been shaken and he was still trying to understand what happened. I do not know if he has since come to any conclusions. But it was clear that prior to the near global disaster, he had never even faintly contemplated the possibility that his beliefs may have been in error. We are all still paying the price for his certainties.
43 Yours will be a generation that that will live through times of more than usual uncertainty.
44 A global transition of power and ideas is underway. Transition to what, no one can yet say. We have no maps and will have to improvise our way forward the best we can. It will be a transition measured in decades and not just a few years, and it is your misfortune that it is occurring as the technology of the internet is making us solipsistic.
45 The internet conflates and confuses our opinion with information and tempts us to immerse ourselves only in a circle of those who share and reinforce our own interests and views. It shortens attention spans and privileges the new and novel over any notion of lasting value. Social media like Facebook have perverted the common meaning of ‘friend’ and ‘like’ beyond all recognition. Only a solipsist or, what is much the same thing, a narcissist, would think that what he or she had for lunch would be of wider interest; and only those with vacuous minds would be interested. And this at a time when the safe navigation of uncharted waters requires a prudent modesty, openness and some minimal capacity for sustained thought.
46 And yet the internet and its associated technologies is indispensible to modern life. We need it to prosper. But what its ultimate effects will be on society, on governance, on international relations, on the very way we think, no one yet knows.
47 I certainly have no answers. As you, the anointed ones, ready yourselves to assume authority and responsibility under these challenging circumstances. I can do no more than to remind you of what Sir Olivier Cromwell wrote to the Synod of the Church of Scotland in 1650. He was trying to persuade the Scots not to embrace the Royalist cause of King Charles the Second and so avert civil war.
48 Gentlemen, he wrote, “I beseech you, in the bowels of Christ” -- and I should explain that in the 17th Century the bowels were considered to be the seat of pity or the gentler emotions -- Gentlemen, Cromwell wrote, “I beseech you, in the bowels of Christ, think it possible that you may be mistaken”.
49 So, Ladies and Gentlemen of the 21st Century, I too beseech you from whatever portion of anatomy you consider most dear, think it possible that you may be mistaken.
50 Before I conclude, you may wish to know how it all ended.
51 Cromwell's advice was not heeded. Shortly thereafter, the third English Civil War broke out. This set in motion a historical trajectory of political, social and economic changes that led to modern Britain, the industrial revolution, the East India Company, Sir Stamford Raffles, the British Empire, the founding of Singapore and ultimately, you and I.
52 And all because good advice fell on deaf ears.
53 What better way to appreciate the irony and contingency of events than to ponder what may have happened if Cromwell's advice was in fact taken and civil war avoided. And as you do so, consider also the possibility that you may be mistaken when you think you are mistaken.
54 And with that final paradox I will end.
55 Thank you for listening to me
[unquote]
wow.
1.5.12
13.4.12
Hangman Strategy
This was taken from "A Better Strategy for Hangman" from DataGenetics.
There’s no easy way to say it. You’ve probably been playing Hangman wrong your entire life! What is the optimal strategy for guessing letters to maximize the chances of getting your first letter? What is hangman? In its purest form, hangman is a word game played between two people. One person selects a secret word, and the other tries to determine the word by guessing it letter-by-letter.
The player with the secret writes a series of dashes, one representing each letter in the solution. Initially, no information is known about the target word, other than its length. The solver calls out letters, one-by-one. If a called letter appears in the solution, all occurrences in the solution are filled in. If the letter does not appear in the solution, the secret writer adds one element to a drawing of a gallows (complete with a stick man).
A complete rendering takes eleven moves. If the hangman drawing gets completed (eleven incorrect letters), then the secret writer has won. If all letters of the word are revealed before this happens, then the solver wins.
As a young person, when you first started to play the game, you probably called out random letters. Once you got a hit of a couple of letters, it helped you narrow down the solution. Next, you probably graduated to calling vowels first, having learned that (just about*) all words contain at least one vowel (or the letter ‘Y’).
* The very complete word dictionary I’m using for this exercise contains 172,806 words. Only twenty of these words do not contain any vowel, or the letter ‘Y’ e.g. CWM, TSKTSK, PSST, PHPHT and BRRR. (I counted just 121 words with that do not contain any of the letters 'AEIOU', so the number that use just 'Y' as a vowel is 101).
Enlightenment - Guessing the first letter
Next, you probably graduated to learning that not all letters are used equally. It’s rare that the letter ‘Q’ appears in a word, whereas ‘T’ is used a lot more often.
Once you get just a couple of letters of in a hangman puzzle, the game becomes easier. The solution set is drastically reduced, and skills like pattern matching and word knowledge become important. It’s crucial to get that first letter in the puzzle as soon as possible. Which letter should you guess first?
Code, Cyphers and Secret Writing
Growing up you probably invented or used your own substitution cypher (where each letter is replaced by a different letter or symbol). A classic example is the Pigpen Cypher. Messages encoded in a simple cypher are pretty easy to crack because the same letter always is always represented by the same symbol. If you solved a lot of puzzle cyphers, then you probably learned and used the letter ordering below.
Ordering of letter frequency in English language: ETAOIN SHRDLU CMFWYP VBGKQJ XZ
The sequence above represents the usage order of letters in the English language, with the letter ‘E’ being the most common letter, followed by the letter ‘T’, all the way down to the letter ‘Z’, the least commonly used.
So, the first letter we should guess when trying to solve a hangman is the letter ‘E’, right?
Since ‘E’ is the most popular letter in English text, it will have the highest probability of being in our word, right?
Wrong!
First mistake
Yes, the ordering above is an accurate portrayal of the frequency of usage of letters in English text, and if we were examining English textit is what we should be using …
… But, we’re not looking at pages of text, we’re looking at isolated words.
English text is full of words that are used very frequently:THE, OF, AND, A, TO, IN, IS, YOU, THAT, IT …
A frequency table of letter usage based on English text is biased because of the substantial presence of these common words.
(About one-third of all printed English material is made up of the top 25 occurring words. The most popular 100 words make up approximately one-half of all printed English!).
Because we’re trying to guess a naked word in isolation the above frequency distribution is not appropriate. It's distorted.
First Refinement
Instead, what we need to look for is the incidence of letters in the words in our dictionary, not the incidence of letters in all English text.
This will give a much better probability estimate for the frequency of letters because it will be unbiased by the frequency of common words.
We can further refine this strategy and do a little better. Since we’re happy if we hit one, or many, letters in our target word we do not want to double count frequency if there is more than one of the same letter in a word. Instead of counting the occurrences of all the letters, we count the number of times a letter is present (one or many) times in each word. Essentially giving a count of, if we select a letter, the number of words that this letter is present in.
We can then sort this list based on the probabilities (count of the number of words that letter is present in). Here are the results: ESIARN TOLCDU PMGHBY FVKWZX QJ
There's a noticeable difference. Here, again, is the distribution based on frequency in English text (for comparison). ETAOIN SHRDLU CMFWYP VBGKQJ XZ
Whilst 'E' is still the most popular letter, the next most popular (based on number of words in the dictionary that contain it), is 'S' and not 'T'. 'T' has been relogated to seventh ordinal position (60.13% of all words in my dictionary have a letter 'S' in them, but only 48.23% of them have a letter 'T').
Next in popularity come two more vowels 'I' and 'A' ('O' having moved further back). 'R' occurs significantly more often in isolated words than it does when biased by the frequency in everyday text.
The ordering of vowels is now 'E I A O U' instead of 'E A O I U'
Interestingly, the least likely letter is now 'J' instead of 'Z'. (There are just 2,463 words in the dictionary that contain the letter 'J' cf. 4,592 with the letter 'X' and 7,028 containing the letter 'Z').
Now that we know the chances of a letter being in any word we can use this new table to select our guesses, right?
Wrong!
Don't forget about the length!
The above distribution has been calculated for all the words in the dictionary. But remember, when playing Hangman, we know the length of the word we are trying to guess. This allows us to further refine our searching.
Below is a table showing the popularity of letters in dictionary words grouped by the length of those words. The most popular letters are at the top of the table, and the the least popular letters at the bottom. To the left are the shorter word lengths, and to the right are the longer ones.
There is no word in the English lanaguage that is 18 letters long and contains the letter J. Similarly, there is no twenty letter word which contains the letter W. T is the most popular consonant in three letter words, and falls in popularity in mid-length words before regaining its popularity at fourteen. Z is never the least popular letter.
O falls in popularity in mid-length words.
OK, so our strategy should be to find the column corresponding to the number of letters in the target word, and start calling down the letters from the top until we get a hit, right?
Computers are far better at filtering and sifting through databases. Once a first letter has been found, this knowledge (letters not present, letter found and the position of this letter), massively reduces the solution set of possible words. Tools like SQL and regular expressions can be quickly applied to find all possible words that match the comb filter built up.
Pre-computed tables are only fine up to a point, after that, they become unmanageable. To paraphrase a famous quote:
"Battle plans are excellent up until the first shot is fired!"
The player with the secret writes a series of dashes, one representing each letter in the solution. Initially, no information is known about the target word, other than its length. The solver calls out letters, one-by-one. If a called letter appears in the solution, all occurrences in the solution are filled in. If the letter does not appear in the solution, the secret writer adds one element to a drawing of a gallows (complete with a stick man).
A complete rendering takes eleven moves. If the hangman drawing gets completed (eleven incorrect letters), then the secret writer has won. If all letters of the word are revealed before this happens, then the solver wins.
As a young person, when you first started to play the game, you probably called out random letters. Once you got a hit of a couple of letters, it helped you narrow down the solution. Next, you probably graduated to calling vowels first, having learned that (just about*) all words contain at least one vowel (or the letter ‘Y’).
* The very complete word dictionary I’m using for this exercise contains 172,806 words. Only twenty of these words do not contain any vowel, or the letter ‘Y’ e.g. CWM, TSKTSK, PSST, PHPHT and BRRR. (I counted just 121 words with that do not contain any of the letters 'AEIOU', so the number that use just 'Y' as a vowel is 101).
Enlightenment - Guessing the first letter
Next, you probably graduated to learning that not all letters are used equally. It’s rare that the letter ‘Q’ appears in a word, whereas ‘T’ is used a lot more often.
Once you get just a couple of letters of in a hangman puzzle, the game becomes easier. The solution set is drastically reduced, and skills like pattern matching and word knowledge become important. It’s crucial to get that first letter in the puzzle as soon as possible. Which letter should you guess first?
Code, Cyphers and Secret Writing
Growing up you probably invented or used your own substitution cypher (where each letter is replaced by a different letter or symbol). A classic example is the Pigpen Cypher. Messages encoded in a simple cypher are pretty easy to crack because the same letter always is always represented by the same symbol. If you solved a lot of puzzle cyphers, then you probably learned and used the letter ordering below.
Ordering of letter frequency in English language: ETAOIN SHRDLU CMFWYP VBGKQJ XZ
The sequence above represents the usage order of letters in the English language, with the letter ‘E’ being the most common letter, followed by the letter ‘T’, all the way down to the letter ‘Z’, the least commonly used.
So, the first letter we should guess when trying to solve a hangman is the letter ‘E’, right?
Since ‘E’ is the most popular letter in English text, it will have the highest probability of being in our word, right?
Wrong!
First mistake
Yes, the ordering above is an accurate portrayal of the frequency of usage of letters in English text, and if we were examining English textit is what we should be using …
… But, we’re not looking at pages of text, we’re looking at isolated words.
English text is full of words that are used very frequently:THE, OF, AND, A, TO, IN, IS, YOU, THAT, IT …
A frequency table of letter usage based on English text is biased because of the substantial presence of these common words.
(About one-third of all printed English material is made up of the top 25 occurring words. The most popular 100 words make up approximately one-half of all printed English!).
Because we’re trying to guess a naked word in isolation the above frequency distribution is not appropriate. It's distorted.
First Refinement
Instead, what we need to look for is the incidence of letters in the words in our dictionary, not the incidence of letters in all English text.
This will give a much better probability estimate for the frequency of letters because it will be unbiased by the frequency of common words.
We can further refine this strategy and do a little better. Since we’re happy if we hit one, or many, letters in our target word we do not want to double count frequency if there is more than one of the same letter in a word. Instead of counting the occurrences of all the letters, we count the number of times a letter is present (one or many) times in each word. Essentially giving a count of, if we select a letter, the number of words that this letter is present in.
We can then sort this list based on the probabilities (count of the number of words that letter is present in). Here are the results: ESIARN TOLCDU PMGHBY FVKWZX QJ
There's a noticeable difference. Here, again, is the distribution based on frequency in English text (for comparison). ETAOIN SHRDLU CMFWYP VBGKQJ XZ
Whilst 'E' is still the most popular letter, the next most popular (based on number of words in the dictionary that contain it), is 'S' and not 'T'. 'T' has been relogated to seventh ordinal position (60.13% of all words in my dictionary have a letter 'S' in them, but only 48.23% of them have a letter 'T').
Next in popularity come two more vowels 'I' and 'A' ('O' having moved further back). 'R' occurs significantly more often in isolated words than it does when biased by the frequency in everyday text.
The ordering of vowels is now 'E I A O U' instead of 'E A O I U'
Interestingly, the least likely letter is now 'J' instead of 'Z'. (There are just 2,463 words in the dictionary that contain the letter 'J' cf. 4,592 with the letter 'X' and 7,028 containing the letter 'Z').
Now that we know the chances of a letter being in any word we can use this new table to select our guesses, right?
Wrong!
Don't forget about the length!
The above distribution has been calculated for all the words in the dictionary. But remember, when playing Hangman, we know the length of the word we are trying to guess. This allows us to further refine our searching.
Below is a table showing the popularity of letters in dictionary words grouped by the length of those words. The most popular letters are at the top of the table, and the the least popular letters at the bottom. To the left are the shorter word lengths, and to the right are the longer ones.
| Length of Word | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | |
| #1 | A | A | A | A | S | E | E | E | E | E | E | E | I | I | I | I | I | I | I | I |
| #2 | I | O | E | E | E | S | S | S | S | I | I | I | E | E | E | E | E | S | E | O |
| #3 | E | O | S | A | A | I | I | I | S | S | S | N | T | T | T | T | E | T | E | |
| #4 | I | I | O | R | R | A | A | R | R | N | N | T | S | N | S | N | T | O | T | |
| #5 | M | T | I | O | I | R | R | A | A | A | T | S | N | S | N | S | O | N | R | |
| #6 | H | S | R | I | O | N | N | N | N | R | A | A | A | O | A | O | N | A | S | |
| #7 | N | U | L | L | L | T | T | T | T | T | R | O | O | A | O | A | R | S | A | |
| #8 | U | P | T | T | N | O | O | O | O | O | O | R | R | R | R | R | A | R | N | |
| #9 | S | R | N | N | T | L | L | L | L | L | L | L | L | L | L | L | L | L | C | |
| #10 | T | N | U | U | D | D | D | C | C | C | C | C | C | C | C | C | C | C | L | |
| #11 | Y | D | D | D | U | U | C | D | D | U | P | P | P | P | P | P | P | P | P | |
| #12 | B | B | P | C | C | C | U | U | U | D | U | U | U | U | U | U | M | M | H | |
| #13 | L | G | M | Y | M | G | G | G | G | P | M | M | M | M | M | M | U | U | U | |
| #14 | P | M | H | P | P | P | M | M | M | M | D | G | D | D | H | H | H | H | M | |
| #15 | X | Y | C | M | G | M | P | P | P | G | G | D | H | H | D | D | D | D | Y | |
| #16 | D | L | B | H | H | H | H | H | H | H | H | H | G | G | Y | G | G | G | D | |
| #17 | F | H | K | G | B | B | B | B | B | B | Y | Y | Y | Y | G | Y | Y | Y | G | |
| #18 | R | W | G | B | Y | Y | Y | Y | Y | Y | B | B | B | B | B | B | B | B | B | |
| #19 | W | F | Y | K | K | F | F | F | F | F | V | V | V | V | V | V | V | V | Z | |
| #20 | G | C | W | F | F | K | K | V | V | V | F | F | F | F | F | F | Z | F | V | |
| #21 | J | K | F | W | W | W | W | K | K | K | Z | Z | Z | Z | Z | Z | F | Z | F | |
| #22 | K | X | V | V | V | V | V | W | W | W | K | X | X | X | X | X | X | X | K | |
| #23 | V | J | Z | Z | Z | Z | Z | Z | Z | W | K | K | W | W | Q | Q | K | X | ||
| #24 | J | Z | X | X | X | X | X | X | X | X | W | W | K | Q | W | W | J | J | ||
| #25 | Z | X | J | J | J | Q | Q | Q | Q | Q | Q | Q | Q | K | J | K | Q | Q | ||
| #26 | Q | Q | Q | Q | Q | J | J | J | J | J | J | J | J | J | K | W | ||||
There are so many fascinating things to point out about this table that I don't know where to start!
There are only two words with one letter! There are no two letter words containing the letter C, Q, V or Z.
From one to four letter words, the most popular letter is A. For five letter words it changes to S, then from six to twelve it is the letter E. From thirteen letters onwards, the most likely letter to be in a word is the letter I.
The letter A starts off as the most popular vowel, but by the time words grow to 15 letter long, it has been relegated to fourth most common vowel.\
There are only two words with one letter! There are no two letter words containing the letter C, Q, V or Z.
From one to four letter words, the most popular letter is A. For five letter words it changes to S, then from six to twelve it is the letter E. From thirteen letters onwards, the most likely letter to be in a word is the letter I.
The letter A starts off as the most popular vowel, but by the time words grow to 15 letter long, it has been relegated to fourth most common vowel.\
There is no word in the English lanaguage that is 18 letters long and contains the letter J. Similarly, there is no twenty letter word which contains the letter W. T is the most popular consonant in three letter words, and falls in popularity in mid-length words before regaining its popularity at fourteen. Z is never the least popular letter.
O falls in popularity in mid-length words.
… plus much more …
Here is the same table again with a splash of color highlighting the vowels.
Here is the same table again with a splash of color highlighting the vowels.
| Length of Word | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | |
| #1 | A | A | A | A | S | E | E | E | E | E | E | E | I | I | I | I | I | I | I | I |
| #2 | I | O | E | E | E | S | S | S | S | I | I | I | E | E | E | E | E | S | E | O |
| #3 | E | O | S | A | A | I | I | I | S | S | S | N | T | T | T | T | E | T | E | |
| #4 | I | I | O | R | R | A | A | R | R | N | N | T | S | N | S | N | T | O | T | |
| #5 | M | T | I | O | I | R | R | A | A | A | T | S | N | S | N | S | O | N | R | |
| #6 | H | S | R | I | O | N | N | N | N | R | A | A | A | O | A | O | N | A | S | |
| #7 | N | U | L | L | L | T | T | T | T | T | R | O | O | A | O | A | R | S | A | |
| #8 | U | P | T | T | N | O | O | O | O | O | O | R | R | R | R | R | A | R | N | |
| #9 | S | R | N | N | T | L | L | L | L | L | L | L | L | L | L | L | L | L | C | |
| #10 | T | N | U | U | D | D | D | C | C | C | C | C | C | C | C | C | C | C | L | |
| #11 | Y | D | D | D | U | U | C | D | D | U | P | P | P | P | P | P | P | P | P | |
| #12 | B | B | P | C | C | C | U | U | U | D | U | U | U | U | U | U | M | M | H | |
| #13 | L | G | M | Y | M | G | G | G | G | P | M | M | M | M | M | M | U | U | U | |
| #14 | P | M | H | P | P | P | M | M | M | M | D | G | D | D | H | H | H | H | M | |
| #15 | X | Y | C | M | G | M | P | P | P | G | G | D | H | H | D | D | D | D | Y | |
| #16 | D | L | B | H | H | H | H | H | H | H | H | H | G | G | Y | G | G | G | D | |
| #17 | F | H | K | G | B | B | B | B | B | B | Y | Y | Y | Y | G | Y | Y | Y | G | |
| #18 | R | W | G | B | Y | Y | Y | Y | Y | Y | B | B | B | B | B | B | B | B | B | |
| #19 | W | F | Y | K | K | F | F | F | F | F | V | V | V | V | V | V | V | V | Z | |
| #20 | G | C | W | F | F | K | K | V | V | V | F | F | F | F | F | F | Z | F | V | |
| #21 | J | K | F | W | W | W | W | K | K | K | Z | Z | Z | Z | Z | Z | F | Z | F | |
| #22 | K | X | V | V | V | V | V | W | W | W | K | X | X | X | X | X | X | X | K | |
| #23 | V | J | Z | Z | Z | Z | Z | Z | Z | W | K | K | W | W | Q | Q | K | X | ||
| #24 | J | Z | X | X | X | X | X | X | X | X | W | W | K | Q | W | W | J | J | ||
| #25 | Z | X | J | J | J | Q | Q | Q | Q | Q | Q | Q | Q | K | J | K | Q | Q | ||
| #26 | Q | Q | Q | Q | Q | J | J | J | J | J | J | J | J | J | K | W | ||||
OK, so our strategy should be to find the column corresponding to the number of letters in the target word, and start calling down the letters from the top until we get a hit, right?
| Wrong! | (Though now we're a lot closer to an optimal strategy!) |
Conditional Probability
Results
Here are the final results of these calculations. These charts tell you what order to call letters, based on length of the word, to maximize your chances of getting your first hit.
| There are some interesting take aways from these results:
|
Carrying on
The above analysis (finding our first letter) is easy to render in table form because there are only two choices: We either miss, or we hit. If we miss, we simply try again. Once we've hit a letter or two, however, things get too complex to display in table format. e.g. "Show me the next best letter to guess for eight letter words that have do not have an 'E' or 'I', but have an 'A' and a 'T'! " We'd have a stack of tables reaching up to the ceiling for all combinations of letters present or not, and their positions!Computers are far better at filtering and sifting through databases. Once a first letter has been found, this knowledge (letters not present, letter found and the position of this letter), massively reduces the solution set of possible words. Tools like SQL and regular expressions can be quickly applied to find all possible words that match the comb filter built up.
Pre-computed tables are only fine up to a point, after that, they become unmanageable. To paraphrase a famous quote:
"Battle plans are excellent up until the first shot is fired!"
Subscribe to:
Posts (Atom)