Hangman Strategy

This was taken from "A Better Strategy for Hangman" from DataGenetics.

There’s no easy way to say it. You’ve probably been playing Hangman wrong your entire life! What is the optimal strategy for guessing letters to maximize the chances of getting your first letter? What is hangman? In its purest form, hangman is a word game played between two people. One person selects a secret word, and the other tries to determine the word by guessing it letter-by-letter.

The player with the secret writes a series of dashes, one representing each letter in the solution. Initially, no information is known about the target word, other than its length. The solver calls out letters, one-by-one. If a called letter appears in the solution, all occurrences in the solution are filled in. If the letter does not appear in the solution, the secret writer adds one element to a drawing of a gallows (complete with a stick man).

A complete rendering takes eleven moves. If the hangman drawing gets completed (eleven incorrect letters), then the secret writer has won. If all letters of the word are revealed before this happens, then the solver wins.

As a young person, when you first started to play the game, you probably called out random letters. Once you got a hit of a couple of letters, it helped you narrow down the solution. Next, you probably graduated to calling vowels first, having learned that (just about*) all words contain at least one vowel (or the letter ‘Y’).

* The very complete word dictionary I’m using for this exercise contains 172,806 words. Only twenty of these words do not contain any vowel, or the letter ‘Y’ e.g. CWM, TSKTSK, PSST, PHPHT and BRRR. (I counted just 121 words with that do not contain any of the letters 'AEIOU', so the number that use just 'Y' as a vowel is 101).

Enlightenment - Guessing the first letter

Next, you probably graduated to learning that not all letters are used equally. It’s rare that the letter ‘Q’ appears in a word, whereas ‘T’ is used a lot more often.

Once you get just a couple of letters of in a hangman puzzle, the game becomes easier. The solution set is drastically reduced, and skills like pattern matching and word knowledge become important. It’s crucial to get that first letter in the puzzle as soon as possible. Which letter should you guess first?
Code, Cyphers and Secret Writing

Growing up you probably invented or used your own substitution cypher (where each letter is replaced by a different letter or symbol). A classic example is the Pigpen Cypher. Messages encoded in a simple cypher are pretty easy to crack because the same letter always is always represented by the same symbol. If you solved a lot of puzzle cyphers, then you probably learned and used the letter ordering below.


Ordering of letter frequency in English language: ETAOIN SHRDLU CMFWYP VBGKQJ XZ

The sequence above represents the usage order of letters in the English language, with the letter ‘E’ being the most common letter, followed by the letter ‘T’, all the way down to the letter ‘Z’, the least commonly used.

So, the first letter we should guess when trying to solve a hangman is the letter ‘E’, right?

Since ‘E’ is the most popular letter in English text, it will have the highest probability of being in our word, right?

Wrong!
First mistake

Yes, the ordering above is an accurate portrayal of the frequency of usage of letters in English text, and if we were examining English textit is what we should be using …

… But, we’re not looking at pages of text, we’re looking at isolated words.


English text is full of words that are used very frequently:THE, OF, AND, A, TO, IN, IS, YOU, THAT, IT …

A frequency table of letter usage based on English text is biased because of the substantial presence of these common words.

(About one-third of all printed English material is made up of the top 25 occurring words. The most popular 100 words make up approximately one-half of all printed English!).

Because we’re trying to guess a naked word in isolation the above frequency distribution is not appropriate. It's distorted.
First Refinement

Instead, what we need to look for is the incidence of letters in the words in our dictionary, not the incidence of letters in all English text.


This will give a much better probability estimate for the frequency of letters because it will be unbiased by the frequency of common words.

We can further refine this strategy and do a little better. Since we’re happy if we hit one, or many, letters in our target word we do not want to double count frequency if there is more than one of the same letter in a word. Instead of counting the occurrences of all the letters, we count the number of times a letter is present (one or many) times in each word. Essentially giving a count of, if we select a letter, the number of words that this letter is present in.

We can then sort this list based on the probabilities (count of the number of words that letter is present in). Here are the results: ESIARN TOLCDU PMGHBY FVKWZX QJ

There's a noticeable difference. Here, again, is the distribution based on frequency in English text (for comparison). ETAOIN SHRDLU CMFWYP VBGKQJ XZ

Whilst 'E' is still the most popular letter, the next most popular (based on number of words in the dictionary that contain it), is 'S' and not 'T'. 'T' has been relogated to seventh ordinal position (60.13% of all words in my dictionary have a letter 'S' in them, but only 48.23% of them have a letter 'T').

Next in popularity come two more vowels 'I' and 'A' ('O' having moved further back). 'R' occurs significantly more often in isolated words than it does when biased by the frequency in everyday text.

The ordering of vowels is now 'E I A O U' instead of 'E A O I U'

Interestingly, the least likely letter is now 'J' instead of 'Z'. (There are just 2,463 words in the dictionary that contain the letter 'J' cf. 4,592 with the letter 'X' and 7,028 containing the letter 'Z').

Now that we know the chances of a letter being in any word we can use this new table to select our guesses, right?

Wrong!
Don't forget about the length!

The above distribution has been calculated for all the words in the dictionary. But remember, when playing Hangman, we know the length of the word we are trying to guess. This allows us to further refine our searching.

Below is a table showing the popularity of letters in dictionary words grouped by the length of those words. The most popular letters are at the top of the table, and the the least popular letters at the bottom. To the left are the shorter word lengths, and to the right are the longer ones.

Length of Word
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
#1	A	A	A	A	S	E	E	E	E	E	E	E	I	I	I	I	I	I	I	I
#2	I	O	E	E	E	S	S	S	S	I	I	I	E	E	E	E	E	S	E	O
#3		E	O	S	A	A	I	I	I	S	S	S	N	T	T	T	T	E	T	E
#4		I	I	O	R	R	A	A	R	R	N	N	T	S	N	S	N	T	O	T
#5		M	T	I	O	I	R	R	A	A	A	T	S	N	S	N	S	O	N	R
#6		H	S	R	I	O	N	N	N	N	R	A	A	A	O	A	O	N	A	S
#7		N	U	L	L	L	T	T	T	T	T	R	O	O	A	O	A	R	S	A
#8		U	P	T	T	N	O	O	O	O	O	O	R	R	R	R	R	A	R	N
#9		S	R	N	N	T	L	L	L	L	L	L	L	L	L	L	L	L	L	C
#10		T	N	U	U	D	D	D	C	C	C	C	C	C	C	C	C	C	C	L
#11		Y	D	D	D	U	U	C	D	D	U	P	P	P	P	P	P	P	P	P
#12		B	B	P	C	C	C	U	U	U	D	U	U	U	U	U	U	M	M	H
#13		L	G	M	Y	M	G	G	G	G	P	M	M	M	M	M	M	U	U	U
#14		P	M	H	P	P	P	M	M	M	M	D	G	D	D	H	H	H	H	M
#15		X	Y	C	M	G	M	P	P	P	G	G	D	H	H	D	D	D	D	Y
#16		D	L	B	H	H	H	H	H	H	H	H	H	G	G	Y	G	G	G	D
#17		F	H	K	G	B	B	B	B	B	B	Y	Y	Y	Y	G	Y	Y	Y	G
#18		R	W	G	B	Y	Y	Y	Y	Y	Y	B	B	B	B	B	B	B	B	B
#19		W	F	Y	K	K	F	F	F	F	F	V	V	V	V	V	V	V	V	Z
#20		G	C	W	F	F	K	K	V	V	V	F	F	F	F	F	F	Z	F	V
#21		J	K	F	W	W	W	W	K	K	K	Z	Z	Z	Z	Z	Z	F	Z	F
#22		K	X	V	V	V	V	V	W	W	W	K	X	X	X	X	X	X	X	K
#23			V	J	Z	Z	Z	Z	Z	Z	Z	W	K	K	W	W	Q	Q	K	X
#24			J	Z	X	X	X	X	X	X	X	X	W	W	K	Q	W	W	J	J
#25			Z	X	J	J	J	Q	Q	Q	Q	Q	Q	Q	Q	K	J	K	Q	Q
#26			Q	Q	Q	Q	Q	J	J	J	J	J	J	J	J	J	K		W

There are so many fascinating things to point out about this table that I don't know where to start!
There are only two words with one letter! There are no two letter words containing the letter C, Q, V or Z.
From one to four letter words, the most popular letter is A. For five letter words it changes to S, then from six to twelve it is the letter E. From thirteen letters onwards, the most likely letter to be in a word is the letter I.
The letter A starts off as the most popular vowel, but by the time words grow to 15 letter long, it has been relegated to fourth most common vowel.\

There is no word in the English lanaguage that is 18 letters long and contains the letter J. Similarly, there is no twenty letter word which contains the letter W. T is the most popular consonant in three letter words, and falls in popularity in mid-length words before regaining its popularity at fourteen. Z is never the least popular letter.
O falls in popularity in mid-length words.

… plus much more …

Here is the same table again with a splash of color highlighting the vowels.

Length of Word
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20
#1	A	A	A	A	S	E	E	E	E	E	E	E	I	I	I	I	I	I	I	I
#2	I	O	E	E	E	S	S	S	S	I	I	I	E	E	E	E	E	S	E	O
#3		E	O	S	A	A	I	I	I	S	S	S	N	T	T	T	T	E	T	E
#4		I	I	O	R	R	A	A	R	R	N	N	T	S	N	S	N	T	O	T
#5		M	T	I	O	I	R	R	A	A	A	T	S	N	S	N	S	O	N	R
#6		H	S	R	I	O	N	N	N	N	R	A	A	A	O	A	O	N	A	S
#7		N	U	L	L	L	T	T	T	T	T	R	O	O	A	O	A	R	S	A
#8		U	P	T	T	N	O	O	O	O	O	O	R	R	R	R	R	A	R	N
#9		S	R	N	N	T	L	L	L	L	L	L	L	L	L	L	L	L	L	C
#10		T	N	U	U	D	D	D	C	C	C	C	C	C	C	C	C	C	C	L
#11		Y	D	D	D	U	U	C	D	D	U	P	P	P	P	P	P	P	P	P
#12		B	B	P	C	C	C	U	U	U	D	U	U	U	U	U	U	M	M	H
#13		L	G	M	Y	M	G	G	G	G	P	M	M	M	M	M	M	U	U	U
#14		P	M	H	P	P	P	M	M	M	M	D	G	D	D	H	H	H	H	M
#15		X	Y	C	M	G	M	P	P	P	G	G	D	H	H	D	D	D	D	Y
#16		D	L	B	H	H	H	H	H	H	H	H	H	G	G	Y	G	G	G	D
#17		F	H	K	G	B	B	B	B	B	B	Y	Y	Y	Y	G	Y	Y	Y	G
#18		R	W	G	B	Y	Y	Y	Y	Y	Y	B	B	B	B	B	B	B	B	B
#19		W	F	Y	K	K	F	F	F	F	F	V	V	V	V	V	V	V	V	Z
#20		G	C	W	F	F	K	K	V	V	V	F	F	F	F	F	F	Z	F	V
#21		J	K	F	W	W	W	W	K	K	K	Z	Z	Z	Z	Z	Z	F	Z	F
#22		K	X	V	V	V	V	V	W	W	W	K	X	X	X	X	X	X	X	K
#23			V	J	Z	Z	Z	Z	Z	Z	Z	W	K	K	W	W	Q	Q	K	X
#24			J	Z	X	X	X	X	X	X	X	X	W	W	K	Q	W	W	J	J
#25			Z	X	J	J	J	Q	Q	Q	Q	Q	Q	Q	Q	K	J	K	Q	Q
#26			Q	Q	Q	Q	Q	J	J	J	J	J	J	J	J	J	K		W

OK, so our strategy should be to find the column corresponding to the number of letters in the target word, and start calling down the letters from the top until we get a hit, right?

Wrong!

(Though now we're a lot closer to an optimal strategy!)

Conditional Probability

Results

Here are the final results of these calculations. These charts tell you what order to call letters, based on length of the word, to maximize your chances of getting your first hit.

Number of letters    Optimal calling order   1 A I 2 A O E I U M B H 3 A E O I U Y H B C K 4 A E O I U Y S B F 5 S E A O I U Y H 6 E A I O U S Y 7 E I A O U S 8 E I A O U 9 E I A O U 10 E I O A U 11 E I O A D 12 E I O A F 13 I E O A 14 I E O 15 I E A 16 I E H 17 I E R 18 I E A 19 I E A 20 I E

There are some interesting take aways from these results: The most challenging (least deterministically obvious) words to guess are three letter words. It can take up to ten guesses before getting a letter on the board! With less than three letters, it gets easier (there are less possible words), and with more than three letters it becomes less likely there will be any words that you cannot find a letter for quickly. For five letter words, the best first guess is the letter S. This is the onlytime a consonant is the most likely first guess letter. For four letter words, the first non-vowel guess is an S, followed by B and then F (remember, these are only called if all preceeding letters have failed to hit). No row contains more than ten guesses, and since a Hangman game takes eleven fails to lose, it is impossible to come up with any English letter word that will fail at Hangman without a single letter appearing on the board (assuming the optimal search strategy above is followed). A should only be your first guess if the word length is four or less letters. If five letters, go for S first. Between six and twelve letters try E and above that you should call I.

Carrying on

The above analysis (finding our first letter) is easy to render in table form because there are only two choices: We either miss, or we hit. If we miss, we simply try again. Once we've hit a letter or two, however, things get too complex to display in table format. e.g. "Show me the next best letter to guess for eight letter words that have do not have an 'E' or 'I', but have an 'A' and a 'T'! " We'd have a stack of tables reaching up to the ceiling for all combinations of letters present or not, and their positions!

Computers are far better at filtering and sifting through databases. Once a first letter has been found, this knowledge (letters not present, letter found and the position of this letter), massively reduces the solution set of possible words. Tools like SQL and regular expressions can be quickly applied to find all possible words that match the comb filter built up.

Pre-computed tables are only fine up to a point, after that, they become unmanageable. To paraphrase a famous quote:

"Battle plans are excellent up until the first shot is fired!"

Marriage Is For Losers

From drkellyflanagan

You can be right, or you can be married; take your pick. I can’t remember who told me that, but I do remember that they were only half-joking. The other half, the serious half, is exceedingly important. This is why.

Many therapists aren’t crazy about doing marital therapy. It’s complicated and messy, and it often feels out of control. In the worst case scenario, the therapist has front row seats to a regularly-scheduled prize fight. But I love to do marital therapy. Why? Maybe I enjoy the work because I keep one simple principle in mind: if marriage is going to work, it needs to become a contest to see which spouse is going to lose the most, and it needs to be a race that goes down to the wire.

When it comes to winning and losing, I think there are three kinds of marriages. In the first kind of marriage, both spouses are competing to win, and it’s a duel to the death. Husbands and wives are armed with a vast arsenal, ranging from fists, to words, to silence. These are the marriages that destroy. Spouses destroy each other, and, in the process, they destroy the peace of their children. In fact, the destruction is so complete that research tells us it is better for children to have divorced parents than warring parents. These marriages account for most of the fifty percent of marriages that fail, and then some. The second kind of marriage is ripe with winning and losing, but the roles are set, and the loser is always the same spouse. These are the truly abusive marriages, the ones in which one spouse dominates, the other submits, and in the process, both husband and wife are stripped of their dignity. These are the marriages of addicts and enablers, tyrants and slaves, and they may be the saddest marriages of all.

But there is a third kind of marriage. The third kind of marriage is not perfect, not even close. But a decision has been made, and two people have decided to love each other to the limit, and to sacrifice the most important thing of all—themselves. In these marriages, losing becomes a way of life, a competition to see who can listen to, care for, serve, forgive, and accept the other the most. The marriage becomes a competition to see who can change in ways that are most healing to the other, to see who can give of themselves in ways that most increase the dignity and strength of the other. These marriages form people who can be small and humble and merciful and loving and peaceful.

And they are revolutionary, in the purest sense of the word.

Because we live in a culture in which losing is the enemy (except in Chicago, where Cubs fans have made it a way of life). We wake up to news stories about domestic disputes gone wrong. Really wrong. We go to workplaces where everyone is battling for the boss’s favor and the next promotion, or we stay at home where the battle for the Legos is just as fierce. Nightly, we watch the talking heads on the cable news networks, trying to win the battle of ideas, although sometimes they seem quite willing to settle for winning the battle of decibels. We fight to have the best stuff, in the best name brands, and when we finally look at each other at the end of the day, we fight, because we are trained to do nothing else. And, usually, we have been trained well. In the worst of cases, we grew up fighting for our very survival, both physically and emotionally. But even in the best of situations, we found ourselves trying to win the competition for our parents’ attention and approval, for our peers’ acceptance, and for the validating stamp of a world with one message: win. And, so, cultivating a marriage in which losing is the mutual norm becomes a radically counter-cultural act. To sit in the marital therapy room is to foment a rebellion.

What do the rebellious marriages look like? Lately, when my blood is bubbling, when I just know I’ve been misunderstood and neglected, and I’m ready to do just about anything to convince and win what I deserve, I try to remember a phone call we recently received from my son’s second grade teacher. She called us one day after school to tell us there had been an incident in gym class. After a fierce athletic competition, in which the prize was the privilege to leave the gym first, my son’s team had lost. The losers were standing by, grumbling and complaining about second-grade-versions of injustice, as the victors filed past. And that’s when my son started to clap. He clapped for the winners as they passed, with a big dopey grin on his face and a smile stretched from one ear of his heart to the other. His startled gym teacher quickly exhorted the rest of his team to follow suit. So, a bunch of second grade losers staged a rebellion, giving a rousing ovation for their victorious peers, and in doing so, embraced the fullness of what it can mean to be a loser. When I’m seething, I try to remember the heart of a boy, a heart that can lose graciously and reach out in affection to the victors.

In marriage, losing is letting go of the need to fix everything for your partner, listening to their darkest parts with a heart ache rather than a solution. It’s being even more present in the painful moments than in the good times. It’s finding ways to be humble and open, even when everything in you says that you’re right and they are wrong. It’s doing what is right and good for your spouse, even when big things need to be sacrificed, like a job, or a relationship, or an ego. It is forgiveness, quickly and voluntarily. It is eliminating anything from your life, even the things you love, if they are keeping you from attending, caring, and serving. It is seeking peace by accepting the healthy but crazy-making things about your partner because, you remember, those were the things you fell in love with in the first place. It is knowing that your spouse will never fully understand you, will never truly love you unconditionally—because they are a broken creature, too—and loving them to the end anyway.

Maybe marriage, when it’s lived by two losers in a household culture of mutual surrender, is just the training we need to walk through this world—a world that wants to chew you up and spit you out—without the constant fear of getting the short end of the stick. Maybe we need to be formed in such a way that winning loses its glamour, that we can sacrifice the competition in favor of people. Maybe what we need, really, is to become a bunch of losers in a world that is being a torn apart by the competition to win. If we did that, maybe we’d be able to sleep a little easier at night, look our loved ones in the eyes, forgive and forget, and clap for the people around us.

I think that in a marriage of losers, a synergy happens and all of life can explode into a kind of rebellion that is brighter than the sun. The really good rebellions, the ones that last and make the world a better place, they are like that, aren’t they? They heal, they restore. They are big, and they shine like the sun. And, like the sun, their gravitational pull is almost irresistible.

The happy secret to better work

this is worth sharing again. "change the lens in your brain"



yea. lens changed.

the chap chai peng strategy

From the date, this is a pretty old post. But trust me, the advice given out here are evergreen. From one chap chai peng fan to another here is something you might enjoy.

[quote]
The Art of Ordering Mixed Vege Rice 28 March, 2010
Posted by dear1dear2 in Dear1, Food & Beverages, Slice of Life.


I am a rice person, so I tend to eat mixed vege rice very frequently. So, let me share with everyone my personal experience of how to maximize the value of a plate of mixed vege rice.


Visit the stall near the end of lunch/dinner time.


Sure, some of the dishes may be sold out, but there will also be plenty of other dishes still available. Since the stall owner can’t keep the dishes until the next lunch/dinner time, it is in his interest to give away larger portions so that he won’t need to deal with leftovers later.


Get your guy colleague to order for you.


This only applies to girls. Stall owners tend to give a smaller portion to girls compared to guys. However, they charge the same price for both. So it is more value for money for a guy to place the order on behalf of the girl so as to receive a larger portion.


Ask for ‘more rice’ instead of ‘add rice’.


Most stall owners will interpret ‘more rice’ as a larger scoop of rice which will result in more rice. This is generally free. However, ‘add rice’ entails adding a second scoop of rice to the first, and this generally cost additional money.


Make sure the stall owner has space to serve.


In many mixed vege stalls, there are usually two or more people serving dishes as the same time. When you are ordering, make sure to order dishes that are near the person serving you so that he won’t have to stretch over his co-worker to reach your dish. If he is stretching to reach your dish, he probably won’t give you more than usual serving since it is that much of a hassle.


Order the meat first.


This is to create a good impression with the stall owner. By ordering the meat first, your are signaling to the stall owner that you will not be ordering the cheapest items on the menu. This puts the owner in a better mood, so he may give you a larger serving.


Order 1 dish at a time and wait until he finished serving.


Point to your first dish, then wait until the stall owner finish serving it before pointing to the next dish. Do not rush him. Often, the stall owner needs time to gauge the portion of food to serve. If you give him sufficient time, it is likely that he will give you a larger than standard portion. Hurrying him results in less than standard portions.


Pause in between ordering dishes.


Pretend to ponder in between dishes. This gives more time for the stall owner to serve your previous dish which likely will result in a larger portion. When he is finished and noticed that you are still pondering overyour next order, he may feel bored and add a few more slices of your previous dish while waiting for you to make your next choice.


Order the same dish twice.


Use this technique very sparingly. After order a dish, pretend to look distracted and point to the same dish again. The stall owner will likely tell you that he has already served that dish. However, occasionally, he may instinctively go through the motion of serving the dish again and make your portion slightly larger, all for the same price.


Order dishes that are about to run out.


It takes practice to estimate, but it is worth picking dishes that are left with about 1.5 to 2 portions. Chances are, that if there is only about 1.5 to 2 portions of the dish left, the stall owner will think ‘what-the-heck’ and just give you everything. The good is that you get extra servings, the bad is that sauces right at the bottom of the tray are very salty.


Order vege dishes that contain meat sauces.


Some vege dishes have some meat in it to bring out the additional flavour. But it is sold at the price of vege becos the meat is usually in small amounts and only present in the sauce. Tofu dishes are an example of such. By ordering such dishes, you are getting some serving of meat for the price of vege. Nice.


Do not order meat dishes that contain vegetables.


These dishes count as meat but is usual half served with vegetables. It is very not worthwhile, do not order it. An example is the stir fried venison with leek. Go for fully meat dishes instead.


Ask for curry sauce.


Many a time, when you ask for curry sauce, some other ingredients in the curry sauce will find its way onto your rice, such as small pieces of chicken or potato. Cabbage curry sauce will probably bring you some cabbages and longbeans. And it’s free.


Ask for meat sauces.


As mentioned earlier, some sauces are cooked with minced meat. It is near impossible to separate the mince meat from the sauce. So if you ask for meat sauces, you will definitely receive some serving of meat with it. And it’s free.


Become a regular customer.


Talk to the stall owner on occasions. With good rapport, the stall owner will recognize you as a regular customer and will naturally give you extra servings the next time you visit.


Hope you have fun at the mixed vege stall. :D


-Dear1


[unquote]


i love rice too