In The Business of building minds

Here is an amazing recent talk with Ben Goertzel and Ed Snowden.

Compared to my view here they express less concern on the existential risk side, but focus on societal issues. It is not a coincidence that the big companies are pushing the AI approach that works with big data.

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus, the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

• I. J. Good

The thing with an intelligence explosion is that there is likely only one ³. Afterward some a super-intelligent agent is making the decisions in the world.

Bostrom (Bostrom, 2014) describes intelligence explosions, their kinetics, the nature of different kinds of superintelligences etc.

• If you think you can outwit it, try playing a round of chess against a chess computer.⁴
• A super intelligence is perfectly, masterfully socially capable and will be capable of manipulating humans. You can already see this happening with chat GPT in my opinion. It can generate persuasive text. It can talk about emotional matters.

• instrumental goal convergence is the concept that some sub-goals are almost always useful no matter what your ultimate goals are.

  For example, resource acquisition, making yourself smarter or self-preservation.

• Because of instrumental goal convergence an unfriendly AI will appear to be friendly at first, lest it is shut down.
• After acquiring desive strategic advantage, it (or the group of people controlling it) takes over the world.
• malignent failure modes, the kind of failure that can only ever happen once include:
  • perverse instantiation - AI follows some goals we put in but not how we want it.
  • infrastructure perfusion - AI decides to do something else with the atoms we care about. For instance converting everything in reach into computronium, a hypothetically optimum configuration of matter for computation.

The 2010 Flash Crash was a rapid and extreme market decline and recovery within a short period, caused by high-frequency trading algorithms responding to a large sell order.

This is a fascinating episode in the deployment and operation of the software. I love Kevlin Henney talking about some of the mess of historical and current software deploys.

Computers allow us to Do the wrong thing fast.

• Algorithms can be implemented correctly and start doing inappropriate things when certain assumptions no longer hold.
• Things can go so fast that the only thing helping you are the preparations. For example, safety switches you build into the system.
• If this is humanity instead of the stock market, we'd be gone.

Artificial neuronal networks (ANN) are sort of an abstract emulation of how neuronal activity.

McCulloch-Pitts neuron (McCulloch and Pitts 1943) has proved to be an incredibly powerful abstraction.

When deep convolutional networks started putting dog faces on top of images around 2015 ⁵, I was thinking this is the same stuff that brains are made up of.

Allegedly, these looked like LCD-induced visuals.

We are reverse engineering the substrate of the mind, not its architecture.⁶ It is like building a computer by simulating transistors.

What we end up with is a second black box right next to the brain. This gives us very little in terms of understanding how intelligence works.⁷

I am not making claims about the limits of ANNs and LLMs. On the contrary, I believe these might produce very generally capable systems very soon.⁸

The trouble is that we don't know how they work. We have little science saying anything about the capabilities and characteristics of ANNs if you scale them up enough. Or if you build agent-like things out of them as I describe further down.

I think we can say almost nothing about the characteristics of an ANN that is say 100x larger than chat GPT.

In recent times, Go programs were thought to be extremely competent. Stuart Russels research group found a sort of exploit that allows humans to beat those. You build some spiral of stones that the program won't detect as a connected group. Because neural nets can't do recursive reasoning, as Minsky and Papert pointed out 50+ years ago (Minsky and Papert 1969).

The important point is not a capability claim. The problem is that ANNs are unpredictable and potentially brittle in extremely nuanced ways.⁹

Chat GPT is a blow torch directed at completing text. I am not sure what happens if you scale a blow torch. It would be naive to assume it will turn out to be fluid or have any of the malleableness and open-ended curiosity of human children.

Maybe you know the Pixar movie Inside Out (2015), where the main character Joy is part of a mind. And the whole movie is Joy's adventure through the mind of the non-main character, a schoolgirl.

It is worth seeing this movie. This marks a milestone in the public understanding of how the mind works. Sure the details of the characters (based on a neuroscientific concept of Base Emotions) etc. is not very accurate or even self-consistent.

For instance, those emotional characters seem to be fully functioning agents with minds by themselves. So how do those minds work? Are there even smaller emotion-characters in their minds ad-infinitum? You have to build a mind out of smaller, dumber, less comprehending pieces in order to explain anything.

But many main ideas coming from Cognitive revolution are in the movie: The mind exists (behaviorism is bonkers) and is somehow made up of smaller resources, agencies, machinery - modules, some of which are innate.

There is a book by Max Harms called Crystal Society that is super cool. (At least the first part here).

This is a bit like Inside Out but better. More scientific and self-consistent and sort of realistic with how scientists might up with how to build a mind.

Maybe you can engineer a mind by zooming into mentation large enough so you can ask a llm to continue the next part of the epic story that is the mind.

In the abstract, this would be read-eval-prompt-loop, or prompt iteration.

(loop [context {}]
  (-> context prompt read eval recur))

The context is presumably some stuff that happened earlier within the mind, maybe some database contents etc.

You are an agent of a mind. I am the mind-engine.
I am a Clojure program.
I ask you for mentation instructions.

Name: Growth
Purpose: Accumulate competence and resources
Last thoughts:
[ "I should suggest a contracting scheme to the rest of the society and we can program contract functions for that in the mind-engine"]

Instruction examples:
[:eval (println "Hello World, from growth.")]
[:tx-data
[{:xt/id :growth/notes-to-self
  :note "I should write a recursive mind-engine and run it via :eval"}]]

what are the next instructions to the mind-engine?

text-davinci coming with something:

[:eval (println "Growth is an ongoing process. I should continue to seek out new opportunities and resources to expand my capabilities.")]
[:tx-data
[{:xt/id :growth/next-steps
  :note "I should research new technologies and methods to increase my efficiency and effectiveness."}]]

There could then be a few design decisions about this architecture. Depending on how much you let the llm drive, you might get different minds similar to how autistic persons are different from neuro-normal people.¹⁰

I am arguing that this is dangerous. We are dangerously close to some agent-like engineered AIs where we have little idea about the kinds of things it might come up with. If the only way you stop this from going rouge is some content filters on the prompts it builds, I think your safety concept is flawed.

Firstly it could run away and potentially be a seed for an uncontrolled intelligence explosion. Secondly, there is the possibility of half-aligned, dirty AI bombs. Engineered using a similar scheme as above. Tasked with disrupting economies, spreading miss information or destroying energy grids. I am certain the current LLM tech would happily truck along, completing the prompts and potentially causing serious harm.

In computer security, you might build naively secure systems that are still vulnerable to Side-channel attacks. An attacker can use the physical properties of the system to gain information about the system. For instance, a timing attack is based on the time of computations, for example when comparing a string of characters the victims password.

If you want to build a provably save AI, you will have to take into account these kinds of nuances in its architecture (Russel, 2019).

For instance, you might build an agent with a decision-making loop and point to your architecture Since decisions are built here, we can know that it won't do x and y, because z is not part of the decision process.

This would be naive without knowing more about neural nets.

For instance, if the ANN is extremely capable, it might manipulate the electronics in nuanced ways, building a second memory storage. Then, it could start sending messages to itself across multiple calculations of the model. It would obtain agency where we thought no agency was possible.

probably the closest we will come to meeting an intelligent alien (Godfrey-Smith, 2016) ¹¹

Remarkably, the majority (around two-thirds) of the octopus's neurons are distributed among its arms, resulting in a highly decentralized nervous system that endows each arm with an impressive degree of autonomy. This arrangement allows each arm to simultaneously process its sensory information and execute motor strategies when tackling tasks or encountering stimuli, reducing the cognitive load on the central brain ¹².

This fact makes me wonder what is the phenomenology of having such autonomous arms. I imagine being able to move an arm in a general direction with some general directive idea kinda, like grap food. And then the arm moves without further voluntary action. Maybe this is a bit like our breathing muscles, which have a deliberate and autonomous mode. Or sometimes right before sleep some of my muscles move around in semi-structured movements completely without deliberate movement control.

Maybe we end up accidentally building a mind with semi-autonomous octopus arms. And some higher-level part of the mind just tries out different configurations of reality like let's see what happens when oxygen is removed from the atmosphere. While some subroutine deals with the edge of manipulating humans to engineer the desired outcome.

Imagine being able to say:

this and that part will have a concept of human biology and will know that oxygen matters to us, and such and such a part that is deciding what to do next, takes these and those things into account.

We are currently not able to say these things.

I think moral realism is true and I think that an AGI if it is the real deal, curious and scientifically humble, will understand ethical truths and act on them.

So I am not worried at all about the real deal. I believe this truly will be the last invention we need ever make.

I imagine such an AGI to be sort of fluid, adaptable and curious.

This would be the other side of the trouble, the transitioning times would have been successfully navigated, and we would be moving forward into the next phase of civilization.

What worries me is the potential single-mindedness and inscrutability of the technology we are developing right now. I think there is a chance that super intelligence would be single-minded in some ways, or maybe brittle in completely unforeseen ways.

Is The Software Industry even capable currently of building solid, bug-free, systems?

I don't see much reason why this is true, and I don't see why you should, either.¹⁴

It is one of the biggest issues that I quarrel with. And I can potentially do something about it as a software engineer.

Most software nowadays is garbage. Mainstream commercial software is broken. Especially web and mobile development is taking on worse and worse complexities. Everything is full of ads, even government or university software is brittle, haunted by game-breaking bugs and breakage at updates. This is getting worse in recent years.

Broken and bloated systems tend to accrue more complexity over time and tend to get harder to work with.

It seems like the same effect is making whole ways of building software worse and worse.

At some point, they won't be able to get anything done anymore. When dealing with the accidental complexity sucks away the creative power that you need to build software.

Now you are generating more of that mediocre code with a generative AI.

Look at how easy it is to make all this boilerplate!

How about figuring out why you need a boilerplate, to begin with? (Succinctness is Power)

Churning out more code quicker?

No.

This is great for the people selling you those auto-complete tools.

Making more garbage code is scary to me.

More bloated code means using more energy means we contribute to climate change.

Bloat is darkness, bloat is uninspired, bloat is a humanitarian issue.

Just today I wanted to click on a Spotify link a friend send me. Doesn't work because it tries to redirect to the app too many retries.

The moment I interact with mainstream software I am getting a stream of hundreds of little things like this that don't work. And if you are interacting with mainstream software you also have this stream of 100 little things.

Maybe the population has by now internalized it is the fault of the person in front of the PC.

It is not your fault. We all fight those battles. It is the software's fault.

Maybe somehow those software vendors have tricked you into thinking it is the complexity of the problem that is the issue. They are wrong.

It is the nonessential complexities that they bring in, in their tools and their design. Commercial software breeds complexity.

There is too little thought, and too little sensibilities for good software.

This is one of the reasons why programmers should learn Unix and Lisp. Because both these technologies bring a deeper sense of how to put things together in an elegant way. Both allow the programmer to develop intuitions and sensibilities about the nature of good software.

Checkout this flip coin website, that I recently made.

This is the best flip coin website on the web because I just made a flip coin website. And then I added little things as icing on the cake.

The power of love and the joy of building toys is more powerful than trying to make money in a business. #no-deadlines.

The fundamental nature of software is to be clean and small and functional. It is broken by the clutter, that crept into recent history, that is spoiling its pristine quiet competencies.^{15 , 16}

This is why Unix is the best operating system because it is relatively uncluttered.

Joyful and solid software comes from noncommercial free and open-source projects.

Hopefully, our future is shaped by technology that comes from this place. With love and collaboration. ¹⁷

Firstly, it would usher in better technology quicker ¹⁸ and secondly, I don't want messy software to be between me and some minor superintelligence mishaps.

One imagines a button to stop what you are doing right now. I am not saying that such a button would fix the control problem, but the funny thing is I don't trust the Software Industry to make an app that allows me to press the button reliably.

I think we need the science that allows us to say things like:

A process with xyz will produce a mind of such and such qualities.

A system needs components one, two and three in order to be capable of doing this and that.

Once we know how intelligence works it will be some small program. And we look back and think what was the big deal?.

We are playing with black boxes and when I put my ear to one, I can hear the ticking.

It is not good that the most capable systems work with big data and big compute. It creates a context where big businesses are throwing money at bigger AIs. We are rushing and racing towards more capable systems and the context is commercial bullshit. Instead of trading with care and building great tools for all of humanity. The current statistical model paradigm is inscrutable, we cannot know the characteristics of up-scaled systems in advance.

Also, we are moving in a world of misinformation, potentially rendering much of the internet useless.

I don't trust big tech to make good software. We need something more similar to the Internet.

The technology that moved humanity forward historically came from hackers with big visions.

I think we need brilliance, the soul and the heart of the computer, to build solid, well-thought-through software that allows us to move safely into the age of spiritual machines.

The great thing about Clojure is the culture of good design and interactively building solid software.

In future blog posts, I will try to do some toy AI programming using Clojure.