Archives for October 2014

The Singularity is coming.

Google’s Secretive DeepMind Startup Unveils a “Neural Turing Machine”
DeepMind has built a neural network that can access an external memory like a conventional Turing machine. The result is a computer that mimics the short-term memory of the human brain.

One of the great challenges of neuroscience is to understand the short-term working memory in the human brain. At the same time, computer scientists would dearly love to reproduce the same kind of memory in silico.

Today, Google’s secretive DeepMind startup, which it bought for $400 million earlier this year, unveils a prototype computer that attempts to mimic some of the properties of the human brain’s short-term working memory. The new computer is a type of neural network that has been adapted to work with an external memory. The result is a computer that learns as it stores memories and can later retrieve them to perform logical tasks beyond those it has been trained to do.

DeepMind’s breakthrough follows a long history of work on short-term memory. In the 1950s, the American cognitive psychologist George Miller carried out one of the more famous experiments in the history of brain science. Miller was interested in the capacity of the human brain’s working memory and set out to measure it with the help of a large number of students who he asked to carry out simple memory tasks.

Miller’s striking conclusion was that the capacity of short-term memory cannot be defined by the amount of information it contains. Instead Miller concluded that the working memory stores information in the form of “chunks” and that it could hold approximately seven of them.

That raises the curious question: what is a chunk? In Miller’s experiments, a chunk could be a single digit such as a 4, a single letter such as a q, a single word or a small group of words that together have some specific meaning. So each chunk can represent anything from a very small amount of information to a hugely complex idea that is equivalent to large amounts of information.

But however much information a single chunk represents, the human brain can store only about seven of them in its working memory.

Here is an example. Consider the following sentence: “This book is a thrilling read with a complex plot and lifelike characters.”

This sentence consists of around seven chunks of information and is clearly manageable for any ordinary reader.

By contrast, try this sentence: “This book about the Roman Empire during the first years of Augustus Caesar’s rein at the end of the Roman Republic, describes the events following the bloody Battle of Actium in 31 BC when the young emperor defeated Mark Antony and Cleopatra by comprehensively outmaneuvering them in a major naval engagement.”

This sentence contains at least 20 chunks. So if you found it more difficult to read, that shouldn’t be a surprise. The human brain has trouble holding this many chunks in its working memory.

In cognitive science, the ability to understand the components of a sentence and store them in the working memory is called variable binding. This is the ability to take a piece of data and assign it to a slot in the memory and to do this repeatedly with data of different length, like chunks.

During the 1990s and 2000s, computer scientists repeatedly attempted to design algorithms, circuits and neural networks that could perform this trick. Such a computer should be able to parse a simple sentence like “Mary spoke to John” by dividing it into its component parts of actor, action and the receiver of the action. So in this case, it would assign the role of actor to Mary, the role of action to the words “spoke to” and the role of receiver of the action to “John.”

It is this task that DeepMind’s work addresses, despite the very limited performance of earlier machines. “Our architecture draws on and potentiates this work,” say Alex Graves, Greg Wayne, and Ivo Danihelka at DeepMind, which is based in London.

They begin by redefining the nature of a neural network. Until now, neural networks have been patterns of interconnected “neurons” which are capable of changing the strength of the interconnections in response to some external input. This is a form of learning that allows them to spot similarities between different inputs.

But the fundamental process of computing contains an important additional element. This is an external memory which can be written to and read from during the course of a computation. In Turing’s famous description of a computer, the memory is the tickertape that passes back and forth through the computer and which stores symbols of various kinds for later processing.

This kind of readable and writable memory is absent in a conventional neural network. So Graves and co have simply added one. This allows the neural network to store variables in its memory and come back to them later to use in a calculation.

This is similar to the way an ordinary computer might put the number 3 and the number 4 inside registers and later add them to make 7. The difference is that the neural network might store more complex patterns of variables representing, for example, the word “Mary.”

Since this form of computing differs in an important way from a conventional neural network, Graves and co give it a new name—they call it a Neural Turing Machine, the first of its kind to have been built. The Neural Turing Machine learns like a conventional neural network using the inputs it receives from the external world but it also learns how to store this information and when to retrieve it.

The DeepMind work involves first constructing the device and then putting it through its paces. Their experiments consist of a number of tests to see whether, having trained a Neural Turing Machine to perform a certain task, it could then extend this ability to bigger or more complex tasks. “For example, we were curious to see if a network that had been trained to copy sequences of length up to 20 could copy a sequence of length 100 with no further training,” say Graves and co.

It turns out that the neural Turing machine learns to copy sequences of lengths up to 20 more or less perfectly. And it then copies sequences of lengths 30 and 50 with very few mistakes. For a sequence of length 120, errors begin to creep in, including one error in which a single term is duplicated and so pushes all of the following terms one step back. “Despite being subjectively close to a correct copy, this leads to a high loss,” say the team.

Although the sequences involved are random, it’s not hard to imagine how they might represent more complex ideas such as “Mary” or “spoke to” or “John.” An important point is that the amount of information these sequences contain is variable, like chunks.

They compare the performance of their Neural Turing Machine with a conventional neural network. The difference is significant. The conventional neural network learns to copy sequences up to length 20 almost perfectly. But when it comes to sequences that are longer than the training data, errors immediately become significant. And its copy of the longest sequence of length 120 is almost unrecognizable compared to the original.

The DeepMind team go on to test the Neural Turing Machine on other tasks. For example, one of these is the equivalent of photocopying: the task is to copy a sequence and then repeat that sequence a specified number of times and end with a predetermined marker. Once again, the Neural Turing Machine significantly outperforms a conventional neural network.

That is an impressive piece of work. “Our experiments demonstrate that [our Neural Turing Machine] is capable of learning simple algorithms from example data and of using these algorithms to generalize well outside its training regime,” say Graves and co.

That is an important step forward that has the potential to make computing machines much more brainlike than ever before. But there is significant work ahead.

In particular, the human brain performs a clever trick to make sense of complex arguments. An interesting question that follows from Miller’s early work is this: if our working memory is only capable of handling seven chunks, how do we make sense of complex arguments in books, for example, that consists of thousands or tens of thousands of chunks?

Miller’s answer is that the brain uses a trick known as a recoding. Let’s go back to our example of the book and add another sentence: “This book is a thrilling read with a complex plot and lifelike characters. It is clearly worth the cover price.”

Once you have read and understood the first sentence, your brain stores those seven chunks in a way that is available as a single chunk in the next sentence. In this second sentence, the pronoun “it” is this single chunk. Our brain automatically knows that “it” means: “the book that is a thrilling read with a complex plot and lifelike characters.” It has recoded the seven earlier chunks into a single chunk.

To Miller, the brain’s ability to recode in this way was one of the keys to artificial intelligence. He believed that until a computer could reproduce this ability, it could never match the performance of the human brain.

Google’s DeepMind has stated that its goal is “solving intelligence.” If this solution is anything like human intelligence, a good test would be to see whether Neural Turing Machines are capable of Miller’s recoding trick.

Ref: : Neural Turing Machines

See the Burning Man Art.

If you’re not familiar with the art work displayed at the Burning Man Festival, you should take a look.

I’ve published another free story on

This story is about a guy hitchhiking through Georgia who ends up being hassled by the police. The name is,

Thank God For M.L.K.

5 Top Regrets of the Dying

| For many years I worked in palliative care. My patients were those who had gone home to die. Some incredibly special times were shared. I was with them for the last three to 12 weeks of their lives.

Life is a choice. It is your life. Choose consciously, choose wisely and choose honestly. Choose happiness.

hourglass running out of sand-top 5 regrets of dying people

Words of wisdom on life from critically ill people realizing their time on earth is running out. — Ross M. Horowitz /Getty Images

People grow a lot when they are faced with their own mortality. I learned never to underestimate someone’scapacity for growth.

Some changes were phenomenal. Each experienced a variety of emotions, as expected: denial, fear, anger, remorse, more denial and eventually acceptance. Yet every single patient found peace before departing. Every one of them.

When questioned about any regrets they had or anything they would do differently, common themes surfaced. Here are the most common five:

1. I wish I’d had the courage to live a life true to myself, not the life others expected of me.

This was the most common regret of all. When people realize that their life is almost over and look back clearly on it, it is easy to see how many dreams have gone unfulfilled. Most people have not honored even half of their dreams and had to die knowing that it was due to choices they’d made, or not made.

It’s important to try to honor at least some of your dreams along the way. It’s too late once you lose your health. Health brings a freedom very few realize, until they no longer have it.

2. I wish I didn’t work so hard.

This came from every male patient I nursed. They missed their children’s youth and their partner’s companionship. Women also spoke of this regret. But as most were from an older generation, many of the female patients had not been breadwinners. All of the men I nursed deeply regretted spending so much of their lives on the treadmill of a work existence.

By simplifying your lifestyle and making conscious choices along the way, it is possible to not need the income that you think you do. And by creating more space in your life, you become happier and more open to new opportunities, ones more suited to your new lifestyle.

3. I wish I’d had the courage to express my feelings.

Many people suppressed their feelings in order to keep peace with others. As a result, they settled for a mediocre existence and never became who they were truly capable of becoming. Many developed illnesses relating to the bitterness and resentment they carried as a result.

We cannot control the reactions of others. However, although people may initially react when you change the way you are by speaking honestly, in the end it raises the relationship to a whole new and healthier level. Either that or it releases the unhealthy relationship from your life. Either way, you win.

4. I wish I had stayed in touch with my friends.

Often they would not truly realize the full benefits of old friends until their dying weeks, and it was not always possible to track them down. Many had become so caught up in their own lives that they had let golden friendships slip by over the years. There were many deep regrets about not giving friendships the time and effort they deserved. Everyone misses their friends when they are dying.

It is common for anyone in a busy lifestyle to let friendships slip. But when you are faced with your approaching death, the physical details of life fall away. People do want to get their financial affairs in order if possible. But it is not money or status that holds the true importance for them. They want to get things in order more for the benefit of those they love. Usually though, they are too ill and weary to ever manage this task. It all comes down to love and relationships in the end. That is all that remains in the final weeks: love and relationships.

5. I wish that I had let myself be happier.

This is a surprisingly common one. Many did not realize until the end that happiness is a choice. They had stayed stuck in old patterns and habits. The so-called comfort of familiarity overflowed into their emotions, as well as their physical lives. Fear of change had them pretending to others, and to themselves, that they were content. When deep within, they longed to laugh properly and have silliness in their life again.

When you are on your deathbed, what others think of you is a long way from your mind. How wonderful to be able to let go and smile again, long before you are dying.


A True Friend

A Free Short Story at

Namuh Visits

A 4000 word story about the creator’s son coming to Earth to discover humans eating there cousins and he rectifies the situation.

Isaac Asimov Mulls “How Do People Get New Ideas?”


Isaac Asimov Mulls “How Do People Get New Ideas?”

Note from Arthur Obermayer, friend of the author:

In 1959, I worked as a scientist at Allied Research Associates in Boston. The company was an MIT spinoff that originally focused on the effects of nuclear weapons on aircraft structures. The company received a contract with the acronym GLIPAR (Guide Line Identification Program for Antimissile Research) from the Advanced Research Projects Agency to elicit the most creative approaches possible for a ballistic missile defense system. The government recognized that no matter how much was spent on improving and expanding current technology, it would remain inadequate. They wanted us and a few other contractors to think “out of the box.”

When I first became involved in the project, I suggested that Isaac Asimov, who was a good friend of mine, would be an appropriate person to participate. He expressed his willingness and came to a few meetings. He eventually decided not to continue, because he did not want to have access to any secret classified information; it would limit his freedom of expression. Before he left, however, he wrote this essay on creativity as his single formal input. This essay was never published or used beyond our small group. When I recently rediscovered it while cleaning out some old files, I recognized that its contents are as broadly relevant today as when he wrote it. It describes not only the creative process and the nature of creative people but also the kind of environment that promotes creativity.


How do people get new ideas?

Presumably, the process of creativity, whatever it is, is essentially the same in all its branches and varieties, so that the evolution of a new art form, a new gadget, a new scientific principle, all involve common factors. We are most interested in the “creation” of a new scientific principle or a new application of an old one, but we can be general here.

One way of investigating the problem is to consider the great ideas of the past and see just how they were generated. Unfortunately, the method of generation is never clear even to the “generators” themselves.

But what if the same earth-shaking idea occurred to two men, simultaneously and independently? Perhaps, the common factors involved would be illuminating. Consider the theory of evolution by natural selection, independently created by Charles Darwin and Alfred Wallace.

There is a great deal in common there. Both traveled to far places, observing strange species of plants and animals and the manner in which they varied from place to place. Both were keenly interested in finding an explanation for this, and both failed until each happened to read Malthus’s “Essay on Population.”

Both then saw how the notion of overpopulation and weeding out (which Malthus had applied to human beings) would fit into the doctrine of evolution by natural selection (if applied to species generally).

Obviously, then, what is needed is not only people with a good background in a particular field, but also people capable of making a connection between item 1 and item 2 which might not ordinarily seem connected.

Undoubtedly in the first half of the 19th century, a great many naturalists had studied the manner in which species were differentiated among themselves. A great many people had read Malthus. Perhaps some both studied species and read Malthus. But what you needed was someone who studied species, read Malthus, and had the ability to make a cross-connection.

That is the crucial point that is the rare characteristic that must be found. Once the cross-connection is made, it becomes obvious. Thomas H. Huxley is supposed to have exclaimed after reading On the Origin of Species, “How stupid of me not to have thought of this.”

But why didn’t he think of it? The history of human thought would make it seem that there is difficulty in thinking of an idea even when all the facts are on the table. Making the cross-connection requires a certain daring. It must, for any cross-connection that does not require daring is performed at once by many and develops not as a “new idea,” but as a mere “corollary of an old idea.”

It is only afterward that a new idea seems reasonable. To begin with, it usually seems unreasonable. It seems the height of unreason to suppose the earth was round instead of flat, or that it moved instead of the sun, or that objects required a force to stop them when in motion, instead of a force to keep them moving, and so on.

A person willing to fly in the face of reason, authority, and common sense must be a person of considerable self-assurance. Since he occurs only rarely, he must seem eccentric (in at least that respect) to the rest of us. A person eccentric in one respect is often eccentric in others. I think this applies to me.

Consequently, the person who is most likely to get new ideas is a person of good background in the field of interest and one who is unconventional in his habits. (To be a crackpot is not, however, enough in itself.)

Once you have the people you want, the next question is: Do you want to bring them together so that they may discuss the problem mutually, or should you inform each of the problem and allow them to work in isolation?

My feeling is that as far as creativity is concerned, isolation is required. The creative person is, in any case, continually working at it. His mind is shuffling his information at all times, even when he is not conscious of it. (The famous example of Kekule working out the structure of benzene in his sleep is well-known.)

The presence of others can only inhibit this process, since creation is embarrassing. For every new good idea you have, there are a hundred, ten thousand foolish ones, which you naturally do not care to display.

Nevertheless, a meeting of such people may be desirable for reasons other than the act of creation itself.

No two people exactly duplicate each other’s mental stores of items. One person may know A and not B, another may know B and not A, and either knowing A and B, both may get the idea—though not necessarily at once or even soon.

Furthermore, the information may not only be of individual items A and B, but even of combinations such as A-B, which in themselves are not significant. However, if one person mentions the unusual combination of A-B and another unusual combination A-C, it may well be that the combination A-B-C, which neither has thought of separately, may yield an answer.

It seems to me then that the purpose of cerebration sessions is not to think up new ideas but to educate the participants in facts and fact-combinations, in theories and vagrant thoughts.

But how to persuade creative people to do so? First and foremost, there must be ease, relaxation, and a general sense of permissiveness. The world in general disapproves of creativity, and to be creative in public is particularly bad. Even to speculate in public is rather worrisome. The individuals must, therefore, have the feeling that the others won’t object.

If a single individual present is unsympathetic to the foolishness that would be bound to go on at such a session, the others would freeze. The unsympathetic individual may be a gold mine of information, but the harm he does will more than compensate for that. It seems necessary to me, then, that all people at a session be willing to sound foolish and listen to others sound foolish. I agree with this 100%

If a single individual present has a much greater reputation than the others, or is more articulate, or has a distinctly more commanding personality, he may well take over the conference and reduce the rest to little more than passive obedience. The individual may himself be extremely useful, but he might as well be put to work solo, for he is neutralizing the rest.

The optimum number of the group would probably not be very high. I should guess that no more than five would be wanted. A larger group might have a larger total supply of information, but there would be the tension of waiting to speak, which can be very frustrating. It would probably be better to have a number of sessions at which the people attending would vary, rather than one session including them all. (This would involve a certain repetition, but even repetition is not in itself undesirable. It is not what people say at these conferences, but what they inspire in each other later on.)

For best purposes, there should be a feeling of informality. Joviality, the use of first names, joking, relaxed kidding are, I think, of the essence—not in themselves, but because they encourage a willingness to be involved in the folly of creativeness. For this purpose I think a meeting in someone’s home or over a dinner table at some restaurant is perhaps more useful than one in a conference room.

Probably more inhibiting than anything else is a feeling of responsibility. The great ideas of the ages have come from people who weren’t paid to have great ideas, but were paid to be teachers or patent clerks or petty officials, or were not paid at all. The great ideas came as side issues.

To feel guilty because one has not earned one’s salary because one has not had a great idea is the surest way, it seems to me, of making it certain that no great idea will come in the next time either.

Yet your company is conducting this cerebration program on government money. To think of congressmen or the general public hearing about scientists fooling around, boondoggling, telling dirty jokes, perhaps, at government expense, is to break into a cold sweat. In fact, the average scientist has enough public conscience not to want to feel he is doing this even if no one finds out.

I would suggest that members at a cerebration session be given sinecure tasks to do—short reports to write, or summaries of their conclusions, or brief answers to suggested problems—and be paid for that; the payment being the fee that would ordinarily be paid for the cerebration session. The cerebration session would then be officially unpaid-for and that, too, would allow considerable relaxation.

I do not think that cerebration sessions can be left unguided. There must be someone in charge who plays a role equivalent to that of a psychoanalyst. A psychoanalyst, as I understand it, by asking the right questions (and except for that interfering as little as possible), gets the patient himself to discuss his past life in such a way as to elicit new understanding of it in his own eyes.

In the same way, a session-arbiter will have to sit there, stirring up the animals, asking the shrewd question, making the necessary comment, bringing them gently back to the point. Since the arbiter will not know which question is shrewd, which comment necessary, and what the point is, his will not be an easy job.

As for “gadgets” designed to elicit creativity, I think these should arise out of the bull sessions themselves. If thoroughly relaxed, free of responsibility, discussing something of interest, and being by nature unconventional, the participants themselves will create devices to stimulate discussion.

Published with permission of Asimov Holdings.

(No title)

Out of This World Sci-Fi Poetry

Is now on sale. Enter coupon code at

Your coupon code is JN92B (not case-sensitive).

New submission spot on Amazon for writers and for readers to get free books.

Seems better than just sending out submissions and wondering if anyone ever read them.

A critique for “A Penis Manologue.”


Out-did yourself with this book! (A Penis Manologue) You delve into subjects so horrifying and unbelievable…Man’s inhumanity towards man.  The cruelty thrust against the masses, by a few demented human beings….actually sub-humans.  I was not aware of some of the things you uncovered, which have been going on thousands upon thousands of years.  Your research is is your writing!!
You should re-submit your book to the instructors at Yavapai College.  If they cannot accept the facts, as outlined in your book, then they are not qualified to be teaching creative writing. Whose feelings or outlook in life, are they trying to protect?