High-Dimensional Computing With Sparse Vectors Using Clojure

A simple distance measure suffices for HDC. For instance, the Hamming Distance counts the bits that disagree

a: [0 0 0 1 0 ]
b: [0 0 1 0 0 ]

hamming distance = 2

Consider that for sparse vectors, the hamming distance is dominated by all the 0's agreeing. The above pair has a distance of 'only' 2, yet is as far apart as possible (k = 5, count-non-zero = 1).

So I take the overlap of non-zero bits and normalize with the sparsity.

(apply
 max
 (let
     [a (hd/->hv)]
     (for [n (range 50000)]
       (hd/similarity a (hd/->hv)))))
0.07

The similarity of 2 random hypervectors is guaranteed to be tiny.

Here, 0.1 <= similarity(a,b) means that a and b are related.

One can also imagine trying to spoof an address decoder neuron, say N = 1.000 and bits are {1,-1} binary and the address vector is dense. If the address is a and the current input ensemble is the unrelated b, the distance between a and b is roughly d(a,b) = N / 2 = 500. Let's say the neuron threshold (distance to address) is d = 400 = N * 0.4, then you would have to spoof d(a,b) - d = 100 input bits in the right direction towards address a. Say you are a stochastic process like unreliable hardware, for instance, random skip rate and random intrinsic firing rate of neurons. Say that you have a coin flip chance to flip a bit, then for the address neuron to misfire for inputs b would mean to throw a coin 100 times correctly, i.e. 0.5¹⁰⁰, which is astronomically unlikely.

(Math/pow (/ 1 2) 100)
7.888609052210118E-31

In other words, if one picks points from H randomly, there is only a tiny region around a for with the address decoder neuron responds. Even if there is a 400-bit (d = N * 0.4) wiggle room for the neuron to be spoofed. It is quite remarkable that 40% of the bits are allowed to be random in the input, and the address decoder can reliably differentiate noise from its address. P. Kanerva: 1,4,3. This is impressive already with N = 1.000, but here N = 10.000.

This holds to a lesser extent for sparse vectors, but the properties of sparse HDVs are still similarly impressive (Z. Yang for a more mathematical summary)²¹.

One of the deep mysteries of neuroscience is that there is a 40% skip rate for neuron firing 5. This is called failure rate in the literature. But perhaps it is a feature, not a bug.

Perhaps it is the amount of skipping the brain gets away with, maximizing energy efficiency.

Or perhaps this roughly makes the address inputs hover right around the detector threshold. Intuitively, this is like rolling the ball on the top of an attractor landscape in a recurrent neuronal net, making it less stuck in any basin.

I am imagining a piano, where the notes are the neuronal ensembles, a good piano will be fluid, and all keys (notes) are easy to compose.

—

Hearing the voice of a person, say 'Bob', always sounds like the same voice, even though the particulars of the sensor data must be quite variable. In Hofstadter's terms, we make an analogy between the instances of hearing Bob's voice. We understand/categorize/make the analogy / analyze the situation / find the essence of the 'hearing Bob's voice situation'. Whatever represents such an internal situation is quite robust to noise. See Biological Remarks On The Robustness of The Address Decoder (appendix)

auto-associative memory: A memory where the address is the same as the content. This is akin to associating a pattern with itself, allowing a pattern to complete. That is retrieving a pattern from a noisy version of the pattern.

If we allow querying with a-prime, which is different, but similar to a. You get a out, with the noise cleaned up.

A cleanup memory component of an HDC/VSA allows us to recover the initial hyper vector (seed vector a) from a noisy a-prime.

A hetero-associative memory is a memory where address and content are different things. For instance in RAM, the address locations are designated pointers, usually numbers in the word length of the computer (32bit, 64bit,…). The content of the address location is an unrelated string of bits. I.e. all possible bit string permutations go together with any location address. Address and location are different, hence hetero associative.

In HDC/VSA a hetero-associative item memory is needed to map from the hyperdimensional domain the the symbolic domain and back. (Usually, one would remember all seed vectors one creates. Then the item memory can be queried an provide the nearest neighbor of the seed vectors).

There is one other thing you can do with a single hyper vector. That is dropping random bits and thereby getting a 'weaker' version of your data.

(defn weaken
"Returns a weakened vector by dropping segment bits from `a`.

This allows one to express the notion of a mix of hypervectors,
where they contribute to different amount.

Weakening the effect of `b` in a superposition:

(let [a (->hv)
      b (->hv)]
  [(similarity a (thin (superposition a (weaken b 0.5))))
   (similarity a (thin (superposition a b)))])
[0.76 0.58]


Mathematically, this is like drawing a line from a to b in hyperspace.
Bundle finds a point exactly between a and b. distance(c,b) ≅ 0.5 ≅ distance(c,a).

With the weaken operation, you can say how far to move from a to b.
Like moving on the line between the points.

This is deterministic.
"
[a drop-ratio])

Instead of dropping randomly, here I let the bits decide which drop. So this is a kind of Context Dependend Thinning.²²

(def a (->hv))
(similarity a (weaken a 0.5))
0.5

This makes it possible to say things like a little bit of, roughly, mostly… See Ambiguity Primitives.

(superposition a b) returns a vector c that is similar to both a and b.

This is typically a elementwise addition, optionally followed by normalization and tie-breaking.

Superposition is commutative and associative. This is also called a sumset operation because it returns a set of the inputs. Just like with a set in programming, it is a data structure representing all the elements without ordering.

Superposition is the same operation that creates a set.

Other names: bundle, possibly, mix, set, union.

The bind of A and B returns a vector C that is dissimilar to both A and B. C can be used to release either A or B, given the other.

It can be used to represent a symbol-value binding.

• Bind distributes over addition

(require '[bennischwerdtner.hd.binary-sparse-segmented :as hd])

(let [a (hd/->seed)
      b (hd/->seed)
      c (hd/->seed)]
  (=
   (hd/bind c (hd/bundle a b))
   (hd/bundle (hd/bind a c) (hd/bind b c))))
true

• Bind preserves distance

;; 'preserves similarity' in this context means that points that are similar before the transformation
;; are also similar after the transformation.
(let [a (hd/->seed)
      b (hd/->seed)
      between-a-and-b (hd/thin (hd/bundle a b))
      c (hd/->seed)]
  [(hd/similarity a b)
   (hd/similarity a between-a-and-b)
   (hd/similarity b between-a-and-b)
   (=
    (hd/similarity (hd/bind a c) (hd/bind b c))
    (hd/similarity a b))
   (=
    (hd/similarity (hd/bind a c) (hd/bind between-a-and-b c))
    (hd/similarity a between-a-and-b))])
[0.0 0.3 0.7 true true]

;; To put this into other terms, C is mapping from one domain to another domain, preserving relationships

• Depending on the implementation, bind is commutative and/or associative.

Here, bind is commutative and associative: (for maximally sparse vectors).

(= (hd/bind (clj->vsa :a) (clj->vsa :b))
   (hd/bind (clj->vsa :b) (clj->vsa :a)))
true

(= (hd/bind
    (clj->vsa :c)
    (hd/bind (clj->vsa :a)
             (clj->vsa :b)))
   (hd/bind (clj->vsa :b)
            (hd/bind
             (clj->vsa :c)
             (clj->vsa :a))))
true

• Like bind, distributes over addition.
• Like bind, preserves distance.
• Distributes over bind.

Here, we permute by rotating the segments blockwise.

(def a (hd/->seed))
(def b (hd/thin (hd/bundle (hd/weaken a 0.25)
                           (hd/->seed))))

;; permuting creates a vector dissimilar to 'a'
(hd/similarity (hd/permute a) a)
0.0

;; permute-inverse is the inverse of permute
(= (hd/permute-inverse (hd/permute a)) a)
true

;; permute-inverse is simply permuting with negative
;; n. It just means rotate the other way
(= (hd/permute-n (hd/permute-n a 2) -2) a)
true

;; permute preserves distance
(hd/similarity a b)
0.37
(hd/similarity (hd/permute a) (hd/permute b))
0.37

;; permute distributes over addition
(= (hd/permute (hd/bundle a b))
   (hd/bundle (hd/permute a) (hd/permute b)))
true
;; permute distributes over bind
(= (hd/permute (hd/bind a b))
   (hd/bind (hd/permute a) (hd/permute b)))
true

• Also called rotate, protect - unprotect
• You can think of it quote - unquote since the permute can be interpreted as a quotation of a symbol.
• We 'map into an unrelated domain'
• Permute, like bind, is said to be randomizing, because the input is not resembling the output.
• It is interesting to note that permute is trivially implemented in circuits, just make wires that represent the permutation between an input and output register.

The humble permute turns out to be a powerful, simple component for building complex datastructures. See Programming In Superposition.

• Everything is a set, complex datastrucutes overlap, if they share structure.
• Intersection, difference, union work for complex datastructures generally.
• Computing with a set (a superposition) is like computing with an element, and the rest comes along for the ride, in parallel.

To go from symbolic domain to hyper domain, clj->vsa* creates a new seed vector, remembers it in an item memory and returns it.

cleanup* serves as an item memory lookup, retrieving the closest match (with a threshold). I.e. hyper domain to symbolic domain.

(hdd/clj->vsa* :a)
#tech.v3.tensor<int8>[10000]
[0 0 0 ... 0 0 0]

(hdd/cleanup* *1)
'(:a)

We might as well interpret a Clojure set in the symbolic domain, returning a set, which is the same as the superposition.

(=
 (hdd/clj->vsa* #{:a :b :c})
 (hdd/set
  (hdd/clj->vsa* :a)
  (hdd/clj->vsa* :b)
  (hdd/clj->vsa* :c))
 (hd/superposition
  (hdd/clj->vsa* :a)
  (hdd/clj->vsa* :b)
  (hdd/clj->vsa* :c)))
true

Set comes with intersection and difference:

(def sun (hdd/clj->vsa* #{:yellow :warm :space-object :round}))
(def banana (hdd/clj->vsa* #{:yellow :food :tasty :fruit :banana-shaped}))

(hdd/cleanup* (hdd/intersection sun banana))
'(:yellow)

(hdd/cleanup*
 (hdd/difference
  (hdd/clj->vsa* #{:ice :cold :hard :water})
  (hdd/clj->vsa* #{:cold :hard :stony :earth})))
'(:ice :water)

The simple associative map is the superposition of its key-value pairs (created by bind):

(=
 (hdd/clj->vsa* {:a :x :b :y :c :z})
 (hd/superposition
  (hd/bind (hdd/clj->vsa* :a)
           (hdd/clj->vsa* :x))
  (hd/bind (hdd/clj->vsa* :b)
           (hdd/clj->vsa* :y))
  (hd/bind (hdd/clj->vsa* :c)
           (hdd/clj->vsa* :z))))
true

The cool thing is that you treat the set as if it were an element:

(hdd/cleanup*
 (hd/unbind
  (hdd/clj->vsa* {:a :x :b :y :c :z})
  (hdd/clj->vsa* :a)))
(:x)

Unbind releases a value from a key value pair, it also releases a value from the suerposition of key value pairs.

In porgraming in superposition, we general treat the whole as if it was an element. Pretend the forest is a tree so to say. 👈

Querying a map with the result of the intersection of something:

;; -------------------

(def fruit-domain (hdd/clj->vsa*
                   {:lemon :yellow :apple :red :orange :orange}))

(hdd/cleanup*
 (hd/unbind
  fruit-domain
  (hdd/intersection sun banana)))
'(:lemon)

It's input is an address word x of length N, here (N = 10.000).

We match against the address-matrix, made from M (M ~ 1e6) hard address locations. These are preconfigured, vectors of length N. In the original, they are made from bits where each is {1, -1} with probability 1:1. But here, they are sparse {0,1}, and we look at the overlap with x, resulting in an overlap (count) vector d. The sparseness of the address matrix makes this an example of an 'intermediate design' SDM7,²⁵

We cut off d at the decoder-threshold yielding the address-activations, representing the hard locations that are on. Observe that a lower = decoder-threshold yields a larger number of addresses. There is an optimal count of address locations (the activation probability). Which is important but I don't discuss it here for brevity.

Write:

Increment the counters in the content matrix C, cap it with a counter-max, at all the overlap positions of the activated address locations and the input word (length U).

If U equal N, this can be used as auto-associative memory.

Read:

Given address location activations, sum up the counters of C, selecting the activated rows. In the original SDM design, take the sign (majority) of the resulting sums and call it the output word.

Here, I added steps to get a sparse segmented binary hyper vector out. This takes a top-k for each segment. If top-k is equal to 1, the outcome is usually maximally sparse - i.e. there is one 'on' bit per segment. If top-k is > 1, then a sumset of outcomes is returned, allowing the user to query for multiple continuations of their query at the same time. See What Is The Meaning of read-top-k > 1?.

In an auto associative usage mode, the user decodes with address a, then writes into the activated locations the content a.

The user can query with a' to clean up back to a.

converged lookup

This can be done iteratively, where each subsequent a'' is even more similar to a. This is a very surprising thing about the SDM. The initial required similarity is called critical distance 4.

This allows one to rapidly probe the memory, saying whether an item is stored, or not. If the iterative outcomes converge (very few steps like 2-3 here), the chance to converge on an actual stored item is astronomically high.

The cognitive connotation is akin to a tip of the tongue situation, where the user is sure they must "know", even though the remembering doesn't work. Conversely, one can extremely quickly and I would say with utter conviction say whether one has any chance of remembering something. One "knows" one knows or knows that one "ought to know" something.

Using libpython-clj and pytorch, I was able to build a personal SDM at the Clojure Repl. This utilizes the pytorch sparse tensor feature, which allows us to have a large amount of hard address locations (M) because address-matrix is allowed to be extremely sparse. And content-matrix is small at small datasets. (T < 10.000).

(M = 1e6, N = 1e4) is handled without a big GPU. (GeForce GTX 1050 Ti, only 4096MiB RAM).

This serves as an example of "neural modeling" using pytorch and libpython-clj.

Code is here: sdm.clj (quite raw, interfaces not figured out and so forth, ad-hoc datatransfers python-cuda-jvm etc.).

• Querying with top-k > 1 results in a set of "possible values" for the query.
• The mechnaism for this is what I call unmasking secondary or latent storage counters, which then contribute to the outcome.
• This is similar to an excitability iceberg [Yuste 2022]. In this case, top-k represents roughly the excitability of purkinje cells.
• This is akin to querying for the halo²⁶ of a neuronal ensemble.
• Roughly: Modifying the address decoder threshold increases the radius hyper sphere, modifying the query top-k is saying how many centers to query for.
• Put in another way, the top-k paramater is like the water we fill into the attractor landscape, which is bounded by the address location.
• Increasing top-k to segment count is like filling the water so the complete landscape is covered, where values in between give intermediate 'amounts of contributing outcomes'.
• I.e. querying with top-k = 3 returns a concept cloud of 3 related hypervectors. Querying with top-k = 1 is akin to querying for an ensemble center.
• In Hebbs terms that is fringes and cores. In Braitenberg terms that is halo and centers.
• In light of Attention Approximates Sparse Distributed Memory, this suggests a possible way of updating transformer architecture, this would correspond to having multiple values per attention key.

(def m (sparse-sdm {:address-count (long 1e5)

                    :address-density 0.00003
                    :word-length (long 1e4)}))

(def d (hd/->seed))
(def d-related (hd/thin (hd/superposition (hd/->seed) d)))

(do
  (write m d d 1)
  (write m d-related d-related 1))


;; top-k = 1 I get something out that is ca 50% d
(hd/similarity d-related
               (torch->jvm (:result (lookup m d 1 1))))
0.75

;; and 100% d
(hd/similarity d
               (torch->jvm (:result (lookup m d 1 1))))
1.0

;; with top-k = 2 you get both of them out,
;; querying with either of them
;;
;; intuitively, this is because they
;; share 50% of their address locations. So
;; those bit counters are high, but not the
;; highest. with top-k > 1, we *unmask* the
;; relatively high
;; ("secondary") bit counters, this returns us
;; the superposition of *related* addresses
;;
(hd/similarity d
               (torch->jvm (:result (lookup m d 2 1))))
1.0
(hd/similarity d-related
               (torch->jvm (:result (lookup m d 2 1))))
1.0
(hd/similarity d-related
               (torch->jvm
                (:result (lookup m d-related 2 1))))
1.0

;; to sanity check, a random seed (not stored)
;; will not result in anything resembling
;; anything else, even if top-k is 'high'
(let [e (hd/->seed)]
  (hd/similarity e
                 (torch->jvm (:result
                              (lookup m e 10 1)))))
0.05

;; but top-k == segment-length actually results
;; in 'everything' (this is akin to epilepsy at the purkinje cells)
(= (let [e (hd/->seed)]
     (hd/similarity e
                    (torch->jvm
                     (:result (lookup m e 500 1)))))
   1.0)
true

(= (let [e (hd/->seed)]
     (dtt/->tensor (torch->jvm (:result
                                (lookup m e 500 1)))
                   :datatype
                   :int8))
   (hd/->ones))
true

Unmasking, roughly:

This is more interesting in a hetero associative usage mode.

(do
  (alter-var-root
   #'hd/default-opts
   #(merge %
           (let [dimensions (long 1e4)
                 segment-count 20]
             {:bsdc-seg/N dimensions
              :bsdc-seg/segment-count segment-count
              :bsdc-seg/segment-length
              (/ dimensions segment-count)})))

  (def m (sparse-sdm {:address-count (long 1e5)
                      :address-density 0.00003
                      :word-length (long 1e4)}))

  (def d (hd/->seed))
  (def d-related (hd/thin (hd/superposition (hd/->seed) d)))

  ;; heteroassociative:
  ;; d->a
  ;; d-related->b

  (def a (hd/->seed))
  (def b (hd/->seed))

  (do
    (write m d a 1)
    (write m d-related b 1))

  ;; sanity check, the address content should be different from the address
  (hd/similarity d
                 (torch->jvm (:result (lookup m d 1 1))))
  0.0

  ;; query with
  ;; top-k = 1

  [:a
   (hd/similarity a (torch->jvm (:result (lookup m d 1 1))))
   :b
   (hd/similarity b (torch->jvm (:result (lookup m d 1 1))))]
  [:a 1.0 :b 0.0]

  ;; top-k = 2
  [:a
   (hd/similarity a (torch->jvm (:result (lookup m d 2 1))))
   :b
   (hd/similarity b (torch->jvm (:result (lookup m d 2 1))))]
  [:a 1.0 :b 1.0]

  ;; ... look I get the superposition of both a and b,
  ;; even though a and b are unrelated, they are possible continuations of `d`
  ;; but `b` is only unmasked secondarily, in a more 'divergent mode' of querying.

  )

• Cognitive connotations: It might help model "possibility clouds", or "concept halos", reminiscent of cognitive fluidity [Mitchel, Hofstadter²⁷].

—

The code comments here, while raw in some ways, reflect a complete thought process of designing a k-fold sequence memory. Arguably, what folllows here is a terser version of the code.

—

A simple way to encode a sequence [a,b,c,d] in an SDM is to encode each pair a->b, b->c and so forth as an address-content pair.

So using the memory in a heteroassiative mode, where the outcome of query 1 is in turn address for query 2 and so forth.

;;
;;      a     +-->   b      +-> c
;;   +-----+  |   +-----+   |
;;   |     |--+   |     |---+    ...
;;   +-----+      +-----+
;;
;;    k = 0  ,     k = 1
;;
;;

• This is akin to a pointer-chain or linked-list.
• This breaks down the moment we share an element x across multiple sequences in the dataset.

• Parallel fibers are slow conducting, this would overlap with the need for a delayed address decoder in a k-fold memory.
• The Cerebellum as a kind of clock [e.g. Braitenberg ²⁸], is the second main "cerebellum theory", but out of favor recently.

See design and implementation.

I reproduce the code comment here:

;; Version 3:
;;
;; K fold memory
;;
;; - In [2] Kanerva is building up the principle with a vocubulary, it is reproduced here, not completely but mostly
;;   and with somewhat different notation that fits the context here:
;;
;; (T is the dataset, like everything the machine encounters).
;;
;; - kth-Order Transition: 'A sequence occuring in xs(T) is a subsequence of xt(T) of length k + 1
;;   For example [a,b,c,d] is a third-order transition; it says that the three-element sequence [a,b,c] is followed
;;   by d.'
;;
;; - j-Step Transition:
;;   A pair of elements in xs(T) separated by j - 1 elements. For example [a,d] is a three-step transition,
;;   it says that a is followed by d.
;;   A first-order transition (here, the 'pointer chain' Version 1) is a one-step transition.
;;   We already see that j-step transitions are easy to store in random access memory.
;;   Store `d` with `a` as address.
;;
;; - j-Step Memory:
;;   A (sparse distributed) memory sdm-j[j] that stores the j-step transitions of xs(T).
;;   The data retrieved from reading with `x` can be denoted data-j[j,x],
;;   or `(lookup-xs sdm-j[j] x)`
;;   It is further assumed that the data become avalable j time steps after the address x has been presented to the memory,
;;   so that the memory has a built-in delay. 👈
;;
;; - k-fold Memory:
;;   A k-fold memory is a set of k j-step memores {sdm-j[1], sdm-j[2], ..., sdm-j[k]}.
;;   The jth memory, sdm-j[j] is also referred to as the jth fold.
;;
;; - k-fold Data:
;;   The k-fold data at time T, D(T), are the multiset of the data available at time T.
;;
;; - k-fold prediction:
;;   To summarize (vaguely):
;;
;;   Consider having read `a` from a one-fold memory:
;;
;;   [ a, ? ],
;;
;;   the data at hand will be `b'`, and the prediction input came solely from the addresses read with `a` at time t - 1.
;;
;;   In a k-fold prediction, reading at
;;
;;   [a,b,c,d,?],
;;
;;   The addresses read t - 1, t - 2, t - 3, t - 4 all contribute to the prediction of `?`
;;   If T includes sequences:
;;
;;   [a,b,c,d,e]  xs1
;;   [j,k,l,d,f]  xs2
;;
;;   Then, the k-prediction will have roughly
;;
;;   1/k contributions from the xs2 for `f`.
;;
;;   1/k * 4 contributions from xs1 for `e`,
;;
;;   The data will include overwhelmingly more of `e`, than `f`, so `e` is the 4-fold prediction for [a,b,c,d,?]
;;

;;
;; ---------------------------------
;; The delayed address decoder:
;; ---------------------------------
;;
;; 1. split all addresses into `k-delay` buckets
;; So that each address has an associated `delay`. (0 < `delay` <= `k-delays`)
;; (this can be done by delaying the input to the address decoder, or its output lines,
;; which would fit the neurophysiology of the parallel fibers - they are slow conducting).
;;
;;
;;
;; decoding:
;;
;; When decoding address `a` at time `t0`,
;; activate address locations `a0` at time `t0`,
;; then `a1` at time `t1` and so forth.
;;
;;
;;
;;          a
;;                       +----------- delay lines     (~ slow conducting parallel fibers?)
;;          |            |
;;          |            |
;;          v            v
;;     +----------+      +-+     +---+                  +---+
;;     | -----    |  ----+-+-----|   |-------------->   |   | a1
;;     |          |      |1|     |   |                  |   |
;;     | -----    |   ---+-+-->  |   | a0               |   |
;;     |          |      |0|     |   |                  |   |
;;     +----------+      | |     |   |                  |   |
;;       address decoder | |     +---+                  +---+
;;        ^              | |       |
;;        |              | |       |
;;        |              +-+       |
;;     addresses       delays      v
;;
;;
;;                             activations
;;                             {a0}                     {a1}  adress-location activation set
;;
;;
;;                             t0                        t1
;;
;;
;; 2. When decoding, the resulting activation set is the union of active locations,
;;    Including the ones activated (1...k) steps in the past.
;;
;;
;;  t0:
;;  decode with `a`
;;
;;         a
;;
;;      +-------+         +---+
;;      |       |         |   |
;;      |    A  +-------> |   |
;;      |       |         |   |
;;      +-^-----+         +---+
;;        |                   ^
;;        |                   |
;;     future state:        { a0 }  activation set (now)
;;     [ a1, a2, a3, ...]
;;
;;
;;     at each times step,
;;     some a-lines will be on in the future
;;
;;
;;
;;
;;  t1:
;;  the a-lines are still on for a while
;;  decode with `b`
;;
;;
;;        b               +---+
;;        |               |   |
;;        v               |   |
;;      [ A ]  ---------> |   |
;;       ^                |   |
;;       |                +---+
;;    future state:          ^
;;   [a2,a3, ...]            |
;;   [b1,b2, ...]         { a1 , b0 }  activation set (now)
;;
;;
;;
;;
;;
;;
;;

• The key insight is that a delayed address decoder allows us to smear out the query across multiple pieces of a sequence.
• Both sequences and auto-associative iterative lookups are shown to converge, if the initial address a' is sufficiently similar, inside the critical distance from stored data a.[3]

• reset can be achieved by querying with nothing or non-sense for ~ 7 steps.

• Figure out how to make hierarchical sequences.
• Perhaps a music analogy will help. Perhaps one encodes nested sequences in terms of something like rhythms. (Like Buzsáki suggests 2)

[Based on wikipedia]: (just as reference)

• The count of Granule cells is a bit of a meme, along the lines of "The brain has 30 billion neurons and 50 billion of those are granule cells".
• 3/4 of the number of neurons are cerebellar granule cells. (Out of the window goes the idea that many neurons are what makes Neocortex smart.)
• Input to granule cells comes from mossy fibers.
• Mossy fiber inputs come roughly from the rest of the brain for present purposes.
• Granule cells outnumber mossy fibers 200 to 1.
• Granule cells make four to five dendrites (so each granule cell receives input from 4-5 mossy fibers).
• Each granule cell makes one parallel fiber axon that makes a T shape and goes straight through the molecular layer.
• The molecular layer is made from orderly slaps of Purkinje cell dendrites.
• Purkinje Cells make a very impressive tree-shaped but slap-like dendrite, and the molecular layer is mostly made out of those.
• Each parallel fiber contacts one of 3-5 Purkinje dendrites it passes and makes a total of 80-100 synaptic connections.
• Each Purkinje dendrite receives 200.000 granule inputs
• The sole output of the Cerebellar Cortex is the Purkinje axons, which happen to be inhibitory, making synapses on deep cerebellar nuclei
• (roughly: Deep cerebellar nuclei go to Thalamus then Cortex).
• The proposed module of the cerebellum is a strip of 1000 Purkinje cells,
• receiving from ~ 100 million parallel fibers.
• outputting to a small cluster of ~ 50 neurons in the deep cerebellar nuclei.
• The parallel fibers are slow conducting, unmyelinated. They are among the thinnest known vertebrate axons.
• And 'represent an extreme anatomical adaptation'.
• There are 2 main ideas for this 1) The pf's are adapted to be compact so that the count of pf->purkinje is optimized, OR/AND 2) the pf's are adapted to make delay lines [Braitenberg 1978], which would mean that the cerebellar cortex is computing something in time.
• (not a timeless calculation, functional programmers will appreciate).
• The second input to the cerebellar cortex comes from the inferior olive, which makes one climbing fiber per Purkinje cell (settles during development).
• The climbing fiber for the Purkinje branches and curls around its dendrite, making a multitude of synapses, it makes the strongest excitatory synapses in the brain.
• Clearly, there is a second input line, which has a 1:1 mapping to the output lines (output lines are Purkinje axons, second input lines are climbing fibers).
• This 1:1 correspondence was what allowed Kanerva to map Cerebellum to SDM. He understood that this was required for memory (input and output 1:1).

—

To map SDM and Cerebellum, imagine that the content matrix is the molecular layer, and imagine that the parallel fiber activations are y.

Observe that in read mode Purkinje dendrite can sum up its inputs from its pf-synapses (~ S). Then, per a given "segment" of Purkinje, some can be active (output word).

In write mode, the climbing fiber can activate a set of Purkinje (input word). Purkinje synapses can be put into a "transient receptive mode" (I made up this word, but I am alluding to complex spikes, see literature), where all of the input modifies the strength of the synapses akin to updating the bit counters. This happens to be negative in Cerebellum, but Purkinje is also inhibitory so I suppose somewhere 2 negatives are canceled out in this particular arrangement.

source code

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:a :b :c] :j)))
[:j :k :l]

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:c :b :a] :z)))
[:z :y :x]

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:c :d :c] :j)))
[:j :k :j]

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:j :k :j] :o)))
[:o :p :o]

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:j :k] :o)))
[:o :p]

(into []
      (map hdd/cleanup
        (what-is-the-abc-that-starts-with-j [:c :b :a] :z)))
[:z :y :x]

source code

(tiny-if-abc-got-changed-to-abd-then-what-happenend-to-jkl
 [:a :b :c]
 [:a :b :d]
 [:j :k :l])
'(:j :k :m)

(tiny-if-abc-got-changed-to-abd-then-what-happenend-to-jkl
 [:b :b :b]
 [:a :b :c]
 [:j :j :j])
'(:i :j :k)

Clojure is a highly regarded, functional Lisp hosted on the JVM. It combines the strengths of classical, symbolic, interactive programming of Lisp and Scheme, with the full power of homoiconisity, extendibility via macros, and "having the language always there".

It inherits minimalism and emphasis on functional programming from Scheme while being practical by fully integrated on its host platforms. Thus making it possible to have a Lisp for virtually all modern use cases thinkable.

It improves on Common Lisp by being implemented in terms of abstractions, with powerful abstraction imprimitives³⁰ not the concrete cons-cell. And by providing a powerful set of default, immutable, persistent data structures, which are deeply utilized in every aspect of Clojure programming, including code, which is expanded to utilize vectors and maps, next to lists. (Clojure code contains [] and {} for binding and destructuring).

Data-oriented programming [Sharvit 2022] ²⁹, centering on Clojures immutable data structures can be seen as a continuation³¹ of dynamic Lisp programming with functional style [e.g. Graham 1993³²].

The compression of dimensionality from address to address decoder is reminiscent of the pointers in semantic pointer architecture [Eliasmith]¹.

In G. Buzsáki 2, a very similar thing is hippocampal trajectories, which are preallocated, rich hub neuronal ensemble sequences of the hippocampus. They are more like snippets of music, put together from a pool of notes.

Roughly, many cognition theories include something like "Low dimensional", low detail, abstract, essences, situations [Hofstadter], categories [Feldman-Barret], analogies, frames [Minsky], prototypes [Kanverva 1], or trajectories [Buzsáki], which must be filled with high dimensional, high detail, concrete "data". Called fillers or terminals at ³³.

Note that "low dimensional" is in quotes, because it is a relative statement.

Hearing the voice of a person, say 'Bob', always sounds like the same voice, even though the particulars of the sensor data must be quite variable. In Hofstadter's terms, we make an analogy between the instances of hearing Bob's voice. We understand/categorize/make the analogy / analyze the situation / find the essence of the 'hearing Bob's voice situation'.

The same thing is still true while a rainy thunderstorm is overhead, and both people must raise their voices to be heard. This fits with our observation of the robustness to noise that the address decoder neuron exhibits.

And further, I can "bring" the voice of my friend Bob "to mind", perhaps not or perhaps even concretely "internally listening" to a snippet of auditory sensor data. Even then, the nature, or essence, or gestalt, or seed vector, or inner marble, or gravitational center of meaning [Dennett], or symbol of Bob's voice is quite distinct and seems to be a perfectly valid concept for the mind to deal with.

This has led many thinkers to the conclusion that whatever represents a Bob voice situation can exist independently of Bob voice sensor data. This internalizing (the opposite of externalizing), this detachment from the outside world, this counterfactual planning, or imagining a possible future, or entertaining an idea, or thinking of a thought, or mental navigation2 of mental content is theorized to be supported via neuronal ensembles (Neuronal Ensembles, 5).

The Cell Assembly Hypothesis one might call conceptual neuroscience going back to Hebb [1949] (and before), roughly saying that ensembles make mental content. This is only useful when we can answer questions like how many ensembles are there?, what timewindow does an ensemble have?, how are ensembles structured? and so forth¹².

In Capgras delusion, the user suffers delusional misidentification. Say Allice suffers an accident and will thenceforth think that her husband Bob is not Bob, but Bob-imposter. Believing something strange like "an alien imposter was installed to replace real-bob". Perhaps Alice lost her marbles, or perhaps there was a hyperdimensional symbol bob, which contributed to the gestalt and identity of her perception of Bob. But bob is no longer activated for some reason. Instead, there is another symbol bob-2. In hyperdimensional computing, without further assumptions than randomness, bob-2 is as different from bob than any other person symbol, indeed any other concept symbol.

It is sometimes said that the first impression of a person counts most. Perhaps because we "allocate" some kind of "prototype" "referring" to a person, and the context somehow ends up contributing to its gestalt, basically forever. Perhaps bob-2 underwent the equivalent of a first impression in Alice's mind.

• It's very technical. I can recommend Trenton Bricken's talk, it also gives a nice overview of SDM in the first place: 6.
• Bricken shows how the overlapping hyperspheres of SDM are approximated by the softmax in Attention.
• Both designs amplify the differences between addresses roughly exponentially.
• This is then corroborated by the fact that trained transformers converge on plausible values for SDM parameters. (In SDM, address activation probability is an important factor, in deciding between noise-signal-capacity tradeoffs.)
• Trained transformers have a beta value for their softmax that roughly optimizes query-noise. I.e. it tolerates noisy queries, trading off memory capacity.
• Attention keys - SDM addreses locations,
• Attention query - SDM address word x,
• Attention values - SDM content, ~S, I guess.
• This is quite remarkable, then SDM was pre-empting "Attention" by ~30 years.
• It's really strange to "learn" your memory via backpropagation.
• It is like trying out what the system would do if it would have memory x.
• It's like filling a USB stick by shaping it's content. I find it strange.
• That the alternative has to do with having a dynamical process that actually uses a memory by storing and retrieving lies at hand.
• This suggests that "Attention" captures some essential properties of the computation of the Cerebellum, and that the power of Transformers comes from some computational trick(s) that the Cerebellum is doing, too.
• Note that so-called cerebellum-like structures are found across the animal kingdom, review. In flies they are Mushroom bodies, Are mushroom bodies cerebellum-like structures?. In electric fish, they enlarge evolutionarily alongside electrosensing. There is also one kind of fish with a gigacerebellum, basically the brain of this fish is made from a big cerebellum. I find this funny.

Von Der Malsburg:

Attractor nets are the data structures of the mind. They form a construction kit for our mind states. They are the Lego blocks of the mind. The chunks of cognitive theory. And the syntax, the grammar under which they combine is: Any combination of coactive networks has to form a stabilized, self-organized, attractor net, that would be, given enough time, stable under self-organization. This is in principle a very simple system.

Vectors of Cognitive AI: Self-Organisation

These are all the same thing;

Attractor nets, subnetworks, population vectors, reverberating chains, neuronal ensembles [Lorente], Cell Assemblies [Hebb].

Lorente de Nó (1938) ³⁴ called them chains or ensembles (I will use the term neuronal ensemble), for reviews: ³⁵,³⁶. These are conceptual recurrent connections in neuronal networks that could be activated on their own, thus shedding light on the mechanism and function of internal activity of the brain.

Donal Hebb [1949] called them Cell Assemblies, in his theory, they are self-sustaining, self-activating subnetworks, and the connections between them are allowed to represent causality. If A goes to B, then B follows A. The cell assemblies together with the proposed local learning rule, Hebbian Learning is called Hebbian Theory. Neurons that fire together, wire together - the slogan is somewhat misleading. It is the neurons that cause each other to fire that wire, it takes 2 timesteps to do that.

The Organisation of Behaviour 1949 is a classic; I recently found that Daniel Dennnet mentioned in his memoir³⁷ how Hebb influenced his thinking.

A Hebbian Cell Assembly is a group of neurons that activate each other. Time step 1 the cyan neuron activates the second neuron then so forth, in a chain. They are in something of a "causal alliance", each causing the rest to fire. In that case, such activity would be self-igniting (an old term that I like) or self-stabilizing. If some of the neurons are active, the whole should be recovered, hence it is auto-associative.

A Hopfield net [Hopfield 1982] is a very simple formalism for recurrent neuronal nets inspired by magnetism, it can be used as an auto-associative memory, when used together with a Hebbian Learning Rule. Hopfield Networks are a special case of Sparse Distributed Memory. Thus, the history of HDC, neuronal nets, connectionism, Parallel Distributed Processing [McClelland, Rumelhart], and related ideas have much overlap, Kanerva reviews: 1.

elements and substrates

Note how we are thinking here on the level of the activity on top of neurons. These are elements that perhaps are better seen as some kind of icebergs that float on top of the substrate the neurons make. When they move, they flow through the neurons, made from activity. When they stay, they are circular, self-sustaining activity, like an activity that carved out a subnetwork, or made pocket in the net; Although they might be spread out in time and space, too. Like high dimensional dominos falling 'clack-clack-tick' around the network. "Neurons working together" doesn't quite capture it in my opinion. The key idea is that the activity creates a new emergent entity, the ensemble. Thinking about the ensemble, we disregard the neurons. Like when thinking about the forest, we disregard the trees.

Now you can regard the forest as a kind of entity; For instance, it sort of has tentacles or fingers of trees growing roughly outside of its edges and so forth (The details come from ecology).

The properties of water come from the randomness of the elements. But water is a very predictable substrate.³⁸ Randomness at the neuronal level is very useful, it makes the ensembles more dynamic. Such a randomness or babble at the bottom comes up in many theories of self-organization and emergence.³⁹

Randomness is an example of something that makes sense on the substrate level, not the element level. I think this 'the properties of the substrate' idea is a useful way to think of emergence.

Braitenberg 1978 ⁴⁰, and G. Palm 1982⁴¹ elaborate on the temporal dynamics (temporal structures) of such assemblies. Thus, perhaps an assembly encoding for a piece of music, or a muscle movement would have dynamic parts, as well as static parts.

heterogeneous assembly

• a heterogeneous assembly has multiple sub-centers. E(A) ("excited A")
• The opposite is a homogenous assembly, where all connections would have the same strength.
• Biological intuition tells us cell assemblies should be heterogeneous as a rule.
• If one would reduce the excitability, one would select one of the sub-assembly centers of A.
• The other way around, if one takes a small assembly A1 and increases the excitability, one ignites E(A1),
• Or A1 + halo(A1).
• The halo is something like the city outskirts of A, somewhat connected, but loosely.
• That is a larger cell assembly with multiple centers.

Thought Pump:

• Activate the halo of the assembly by increasing the excitability.
• By decreasing the excitability, the assembly would narrow down into one of its centers.
• This depends on a threshold device, something that regulates the thresholds (excitability) of all neurons in question "area"?, column?, whole cortex?, 'sheet wise', claustrum?, basal ganglia? thalamus?, not figured out.
• It should select the best-connected sub-center (given ideal inhibition, see below).
• This move then, could be done at the speed of 2 neuron timesteps, and it could be a parallel search process.
• Jumping from thought to thought, creating a 'train of thought', hence 'thought pump'.

• If the assemblies are 'settled' and "don't move" anymore, perhaps this could be interpreted as the 'best reachable interpretation' for the system.
• This remains to be purely conceptual afaik. But is a cool idea to have in any case.

Since you are scrolling through already, check out this work on visualizing much of the activity in a zebrafish larva:

Zebrafish larvae are translucent for the first time in their life.

Scientists at the Howard Hughes Medical Institute studied live zebrafish larvae that had been genetically encoded with a calcium indicator called GCaMP5G. They suspended the larva in a gel and then beamed it with lasers. Just before a neuron fires, its action potential is expressed via a spike in calcium ions, so when one of the genetically modified larva's neurons reached its action potential, it glowed. This showed the researchers the firing of the neurons without them having to attach a bunch of electrodes to the fish. Over the course of an hour the researchers used laser beams to scan the larva every 1.3 seconds, exciting the retina of the zebrafish with each scan. This microscopy method allowed the researchers to record up to 80 percent of the fish's 100,000 neurons at single-cell resolution. This is the first time scientists have recorded such a high percentage of an organism's brain activity at such a high resolution.

I'm not saying it makes thought pumps, but something incredibly dynamic is happening either way.

—

In Braitenberg 1978 ⁴² I feel like they become animals made from 'patterns', dynamically dancing, flowing, stretching, like amebas, in the rhythms of music, carrying information, working together to activate remote associations and things like that.

It is a small step to consider them little software animals. When they are information that reproduces, they become memes. If their effect is to be active again, they have causal power or competence without comprehension [Dennett].

Moshes Abeles 1982 was one of the first to do neurophysiology and find something like ensembles (note they were entirely conceptual up to this point!), his theory is dubbed Synfire chain because he and his collaborators emphasize how it is hard for the activity to survive on its own, hence the activity of synchronous neurons could carry forward in a net. Like a forest fire but the forest is sort of wet. And then you have groups of activated neurons going in a chain, or back and forth.

Papadimitriou, et.al. describe a model using Hebbbian Learning⁴³. They proved it to be Turing Complete. (This is considered another candidate for programming neuromorphic hardware.) These are very Hebbian Assemblies. With sheet-inhibition (implemented by a simple cap-k) and a recurrent neuronal net and Hebbian plasticity. From this emerge assemblies, which are auto-associative. They have coherently activated subnetworks of neurons, i.e. attract subnetworks.

For learning, just like with a Hopfield net, you present a sensory stimulus, which you must project or embed into your neurons.

Then, by presenting a part, the whole can be recovered, hence auto-associative.

By presenting 2 stimuli, an overlap assembly will form. By presenting a sequence, hetero associative assembly(s) will form. I imagine this as sort of a tunnel that an assembly with temporal structure takes:

A -> B -> C If A has many connections to B, then B will fire after A. This causality notion is a major thing coming from Hebb's theory.

Such a flow of activity is akin to water flowing through a riverbed. If it goes in a circle, this would also be called a ring attractor.

Essentially, such Hebbian auto-associative subnetworks were already discussed in G. Palm 1982 ⁴¹. Nothing is new under the sun.

—

Rafael Yuste about the electrophysiology of the neuronal ensembles. They are not Hebbian in Cortex

Summarizing:

• They can artificially produce ensembles via opto genetics, it is as if neurons that fire together are glued together (for days!).
• They can make a mouse behave as if it were seeing x, by activating the same ensemble as when it was seeing x. (suggesting ensembles indeed are internal units of perception).
• They can activate ensembles via single neuron activation (remarkable I think). But this only works 60% of the time.
• They find that cortical ensembles are not Hebbian⁴⁴. Not fire together wire together, but everybody active has more excitability. (It's an even simpler, even more local rule, hence very satisfying).
• The important lesson: the causality structure encoding things (the main point of Hebbian Theory) of the brain comes from something else.

—

György Buzsáki¹² provides a cleaned-up definition. The reader-centric cell assembly. His neurophilosophy emphasizes truly satisfying, mechanistic, first principles of thinking. Buzsáki initially wanting to be a radio engineer, sheds light on his work. There are oscillators, rhythms, syntaxes, reader frames, analogies to music. Quite captivating.

Buzsáki's empirical work suggests that we are endowed with pre-allocated symbols, and that nothing is new to the brain, we match our internal symbols (which are more like snippets of music) to the trajectories of our actions and the world.

Buzsáki also suggests certain departures from current neuronal modeling that I am a fan of. The brain is primarily maintaining its internal dynamic. Learning doesn't much change those patterns. This does away from the (essentially behaviorist) notion of the brain as a receptacle of knowledge. Knowledge creation is an intrinsic, self-generated activity.

The primacy of action - The first nervous systems simply moved bodies around and sensors are a secondary invention.

Neurons that wire together fire together.

The meaning of this is that during development, sub-networks of neurons are allocated by growing in the same cohort (~ same day). These subnetworks are stable cell assemblies of the brain - mostly pre-allocated. Like the keys of a piano, it's their composability that matters. This also fits the notion that the brain is a robust system.

The brain is a spaghetti.

Action is the only second opinion. - A cognitive system can ground its internal symbols via action. (How to do this in machine intelligence is a further question, part of the motivation to pursue such things as Sequence Memory - Higher Order Via K-fold, Kanerva also has a shot at a design in 3).

This inside-out philosophy is an answer to David Deutsch's critique of AI (which he says is virtually the oposite of AGI), the idea must be first. The ideas must be essentially random, generic or 'for-free' at the bottom; Tying it together with Kanvervas path of least assumption, asking what you get from assuming nothing but randomness.

More on this in the future, as I want my AI to be essentially inside-out.

This was an early experiment, on the shelf now.

This a sequence processor where all expressions are hypervectors. Where mix, bind etc. are primitive operations. Where symbols are multisymbols and multiple things are evaluated when a multisymbol is encountered.

(cleanup*
  (h-eval (h-read-code (mix :heads :tails))))
 (:heads :tails)

(mix :heads :tails) is the superposition of it's inputs.

Symbols can be multisymbols. If the evaluator encounters a multisymbol in operator position, it will evaluate all possible branches, and return the superposition of the outcomes:

(cleanup* (h-eval (h-read-code ((mix + - *) 10 10))))
;(100 0 20)

similarly, if evaluates multiple branches:

 (cleanup*
  (h-eval
   (clj->vsa
    [['lambda []
      ['if [possibly true false]
       :heads :tails]]])))
; (:heads :tails)

A slightly more substantial piece of hyper lisp code:

(cleanup*
 (h-eval
  (clj->vsa
   ['let ['outcome
          [['lambda []
            ['if [possibly true false]
             :heads :tails]]]]
    [impossibly 'outcome :tails]])))
;;  (:heads)


;; the reader cleans up the need for quoting all the symbols:
(cleanup*
 (h-eval (h-read-code
          (let [outcome ((lambda ()
                                 (if (possibly true false)
                                   :heads
                                   :tails)))]
            (impossibly outcome :tails)))))
;; (:heads)

• Eventually, perhaps we want to program with concepts, or analogies. Vocabulary ideas:
• tolerate - express a range of possible filler values, or express a range of tolerated properties.
• For instance, ballness (the ball frame) tolerates any kind of material, or color and so forth.
• require - express that a property must hold for the fillers, for instance, the shape of a ball is perhaps required to be approximately round.
• Everything other than having a round shape moves away from the notion of baldness in concept space.
• In terms of Mitchell and Hofstadter 1988 such a moving away constitutes /pressure on a concept, which might slip into another concept. This slippage and fluidity is a hallmark of creative cognition.
• A cognitive system might acquire such properties via matching actions and predictions:
• For instance, you might expect to trace the outline of a ball by moving the eyes in a circle, the ball to be graspable, rollable, throwable, be able to be covered by fingers, have a certain weight when held, have a certain distance when reaching out for it, have the same color from all sides, have a texture when stroked, throw a shadow when illuminated by a light source, sink or float in water (depending on its material), be called a 'ball' by other persons, be permissible to fulfill the role of 'ball' in the rules of games that require a 'ball' (or not, then interestingly so), be able to be put into the mouth, be bouncy or fragile depending on the material, be able to be brought to mind from memory, be able to be talked about, be able to have its properties listed, be different from a banana and from almost all other things for that matter, be able to be in the possession of a person
• This list tries to stay within "one degree of complexity", we can easily imagine a ball to be food by ball-eating goblins, to explode in a world where all balls are stuffed with gunpowder, to be worth a million dollars because it is a rare special ball relic of an ancient alien intergalactic soccer game, to be a dragon egg in a world where dragons hide their eggs disguised as ordinary balls
• This being able to put together ideas and concepts must be a deep aspect of computational building materials out of which ideas can be made.
• If any of these properties are stretched too far, for instance, the ball is made from jelly and doesn't roll anymore, then it's not a ball anymore.
• Perhaps a useful hyperlisp would be programming with counterfactuals [inspired by Charia Marletto and David Deutsch's 'Constructor theory'], with what if/s, /possibilities and impossibilities.
• Aaron Sloman [in personal communication]: impossibility, necessary connection, possibility spaces.
• (although he would argue there might be chemical computing at the synapses computing this for all we know).

• G. Buzsáki[2] points out that encoding in time, doesn't need to mean that something in time is being represented.
• A cognitive composite data structure of the brain might be encoded in something that is spread across time.
• In fact, Buzsáki thinks that neuronal syntax works by juxtaposing neuronal letters.
• The putative neuronal letter is an assembly firing in gamma frequency (Gamma waves), juxtaposed in time into neuronal words, that are 7 +/- 2 gamma timestep long (theta frequency).
• In software speech, the elementary or atomic datatype is an ensemble in gamma frequency. A first-order composite is a "theta word" of ~7 elements, (then higher-order composites are made from multiples again and so forth, forming a kind of rhythm hierarchy).
• The Magical Number Seven, Plus or Minus Two is a 'meta' empirical finding of cognitive psychology [Goerge Miller 1956].
• This would mean that the neuronal code⁴⁵ is encoded in time.
• In other words, a neuronal interpreter would read a data structure spread across time.
• To be precise what I have in mind: Imagine you try to build a hyperdimensional ALU using neurons as input and output registers, implementing something like primitive superposition or bind operations, what are the input registers? If I need 2 inputs for a bind (I need 2 input registers), then what is the relationship between the inputs? I'm sure there is a range of possible answers…
• Suppose the inputs are neuronal activity juxtaposed across two subnetworks (maybe the first idea?), shouldn't these be kept in tandem or something? Intuitively, everything that subnetwork A is saying, I think B should be able to say, too. This seems a very strange requirement for brain neuronal net architecture.
• This problem is solved, when the input registers are the same neurons, but juxtoposed in time.
• The relationship is now 1:1. For instance, the overlap of A and B is now simple, it just means overlap.
• This would mean the brain computation is 'rhythmical' or 'musical'. [short talk;⁴⁶, book-length: ⁴⁷, review: ⁴⁸], [Abeles e.g. Spatiotemporal Structure of Cortical Activity: Properties and Behavioral Relevance], [Earl K. Miller].
• If true, then Cognition uses a musical computation; I consider this a joyful twist, that suddenly puts the human taste for music right at the fabric of cognition.
• This might shed light on things like Flash lag illusion.
• If I might speculate, I would think the situation interpretation, ~ "essence finding" in terms of Hofstadter + Sander [2013]²³, has something of a time window (perhaps 1 theta rhythm or 1/2th?), for finding the matching hippocampal trajectory (in terms of Buzsáki[2]), ~ that is roughly one sequential data structure coming out of an SDM here with 7 +/- 2 elements. (Assuming that the Hippocampus is roughly an SDM) - a neuronal word.
• I.e. there would be a 125 - 250 ms time window for finding an interpretation of what "is in short-term memory".

Making use of time:

• Using time for binding is an old idea [C von der Malsburg 1981, review, Abeles 1982]. (Bienstock 1996 ties it together with compositionality in a cool way.)
• An example of making a time-space tradeoff is this neuromorphic, spiking hardware implementation of the block code bind ⁴⁹.
• This works via delay lines, similar to "polychronization feature binding" ⁵⁰.
• Note that it doesn't mean that we (software engineers) need to encode something across time. We merely need to extract useful essential concepts.

Tom Riddle: [Adjusting ring on his finger, the same one in present day Dumbledore's office] Can you only split the soul once? For instance, isn't seven…

Horace Slughorn: Seven? Merlin's beard, Tom! Isn't it bad enough to consider killing one person? To rip the soul into seven pieces… This is all hypothetical, isn't it, Tom? All academic?

Tom Riddle: [Smiling] Of course, sir. It'll be our little secret.

(from Harry Potter Half-Blood Prince).

P. Kanerva, “Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors,” Cognitive Computation, vol. 1, no. 2, pp. 139–159, 2009.

Here and in 3 opens up a new space between computer science and neuroscience. Highly technical, but still accessible. This is truly fascinating work.

Buzsáki, G. (2019). The Brain From Inside Out. New York: Oxford University.

Sparse Distributed Memory, Pentti Kanerva 1988

One might check 1 first. And if one has a taste for more, SDM will be very interesting.

Laiho et.al. 2015 High-Dimensional Computing with Sparse Vectors https://www.researchgate.net/publication/299535938_High-Dimensional_Computing_with_Sparse_Vectors

Rafael Yuste Lectures In Neuroscience, 2023

Kanerva talks:

• Stanford Seminar - Computing with High-Dimensional Vectors
• Pentti Kanerva
• Pentti Kanerva - Modeling the brains computing with hyperdimensional computing
• Pentti Kanerva - The computer and the brain
• Pentti Kanerva - Cerebellum as a neural RAM (UC Berkeley, Fall 2010)
• Pentti Kanerva - Computing in Superposition in the Classical Domain
• Playlist: Some talks
• Related: The Kanerva Machine (went over my head at this point in time).

Attention Approximates Sparse Distributed Memory

Pentti Kanerva 1993 Sparse Distributed Memory and Related Models

https://redwood.berkeley.edu/wp-content/uploads/2020/08/KanervaP_SDMrelated_models1993.pdf

Vector Symbolic Architectures as a Computing Framework for Emerging Hardware Denis Kleyko, Mike Davies, E. Paxon Frady, Pentti Kanerva, Spencer J. Kent, Bruno A. Olshausen, Evgeny Osipov, Jan M. Rabaey, Dmitri A. Rachkovskij, Abbas Rahimi, Friedrich T. Sommer

https://arxiv.org/abs/2106.05268