Towards a Meta Design of LLM Skills

I approach skills philosophically, not as a collection of instructions. That gives layers:

• Layer 0: the debugging moves that are already in the model.
• Layer 1: skills that steer these moves, for example the layer analysis.
• Layer 2: higher-order skills that delegate which skill fits the problem.
• Layer 3: meta skills that apply the same debugging moves to the skills themselves.

The model bisects, minimizes, compares the running with the non-running case and reads error messages without being told. Presumably these moves are already in there from the training data. I cannot prove that from the inside; it is a conjecture that fits what I observe.

So a skill does not have to teach these moves. What is left for it: give the model a direction (what to aim at), enforce completeness (all layers, all siblings) and make what the model claims checkable from the outside.

The layer analysis is a good example, because it steers the model to the design of the software. Without it, a fix easily stays on the surface, one line at the spot where it hurts. With it, you force the model to ask about class, assumption and process gap, and you get deep corrections instead of band-aids. I have not tested whether the model would have stayed at the line without the skill. An ablation (the same case with and without the skill) is still to do.

An expression new Date() returned the real wall-clock time of the machine, where the fixed system time of the evaluation was expected. The fix was one line.

The analysis afterwards needed four levels.

• L1, same mechanism. It was not just the clock. I searched all host imports, globals and library functions and found twelve sources of ambient state: randomness, time zone, order of hash maps, stack depth and more.
• L2, broken assumption. Caches and incremental runs trusted the declared set of inputs. An implicit input made all of them silently wrong. The test "incremental = full run" could not find that, because both runs saw the same wall clock.
• L3, process. The concept text said "same engine, same behavior". There was never a test for that. Same bytes are not the same inputs. Where one test per claim existed (sandbox escape), there was no bug.
• L4 I checked and dropped. It found nothing that L3 did not already have.

Each level got its own countermeasure: a fix, a construction with a checklist, a property test with deliberately shifted ambient state, a table "claim → test".

The point is not that there are four levels. Each level is used to keep searching, and each instance found is checked. A level that merely restates the known bug more abstractly is dropped.

Climb from the concrete to the class, search on every level, name on every level what we did wrong and decide a countermeasure: that is a thinking move. I have captured it as a skill, bug-layer-analysis.

Pólya worked exactly like this for mathematics. In How to Solve It he collects moves with a name and a guiding question, such as specialization, generalization and working backwards ("Could you solve a more general problem? A more special one?"). [Pólya]¹ A skill is, in this sense, an executed heuristic.

Three terms I use for this:

• Thinking object (Denkobjekt): anything you can handle with thought. A bug, a decision, an assumption, a diff. A skill too.
• Thinking operator (Denkoperator): the class of thinking objects that process other thinking objects. The layer analysis takes a defect and returns a class of errors.
• Space of thinking operators: operators are not scattered, they sit in a space with axes. (Provisional, see below.)

The note on the date bug starts after the localization. An agent had found the bug in a case study. How is written nowhere.

Before that come the moves of layer 0: read the error message, reproduce the bug at the push of a button, minimize it, bisect (in space and in time), compare the running with the non-running case, take "that cannot happen" as a signal for a violated assumption. The model does these on its own.

The layer analysis, on the other hand, it does not do on its own. It comes after the fix and asks what the bug says about the design. My bet is that this is exactly the place where a skill changes the most. That is a hypothesis. Evidence would be an ablation and a look into real debugging sessions: where does the model stop without the skill?

From above. I break an operator into axes (object, time, abstraction, stance) and flip one. That is how I got from the layer analysis to the pre-mortem (time arrow flipped) and to the assumption stress test (direction of abstraction flipped).

From below. I ask what someone actually thinks and does when debugging, and derive the operators from that. The technical terms for this are protocol analysis (thinking aloud as a data source) and cognitive task analysis. ²

I took the first way first, and the result is the most honest place in this post: I had four skills that were coherent in themselves and never ran. Placing them in the axes confirmed the construction, because it came from the construction. Then I applied the layer analysis to this error:

• L1: Artifacts are generated from a structure and not tested against practice.
• L2: "A free point in the space is a useful operator" is an assumption, not a finding.
• L3: There is no acceptance criterion for skills. Generating costs nothing.
• Countermeasure: An operator is only built when an observed instance supports it. Every skill carries a section "evidence" with a deletion rule.

That is why I now take the second way. The first is good for seeing gaps, not for deciding what gets built.

Layer 2 delegates. A higher-order skill chooses which skill fits the problem (a router). Schoenfeld showed in empirical studies that Pólya's heuristics alone are not enough. What matters is control: when which strategy is chosen, how progress is monitored and when a path is abandoned. Selection is only one part of that. ³

Layer 3 applies the same debugging moves to the skills themselves. A skill has bugs like code: it can fire on the wrong cases, overlap with another skill or never find anything. Then layer analysis, pre-mortem and stress test can be pointed at it.

If operators are thinking objects, you can apply operators to operators. "Which skills sit next to this one?" is the same move as the layer analysis, one level higher: from the instance to the class, from there to the siblings. That is also the move this post came from.

So far I see five:

• choose an axis,
• invert an operator,
• compose several (layer analysis → L2 assumption → stress test),
• apply an operator to itself,
• choose the fitting one before applying (a router).

The risk is a tower. Meta levels move the problem upwards, and a router can be wrong itself. So I keep the meta level thin: a neighborhood search and a router, nothing more.

The assignment is my judgment, and the axes are chosen, not given. According to Popper, creativity has no logic and therefore no complete list of axes either: a new concept can bring a new axis at any time. A move that fits no axis is the normal case and says that an axis is to be added. Every point in the space is a possible growth point. (I had attributed that sentence to Wheeler and found no source, so it stands here as my thesis.) The same holds for stances and habits of thought.

Debugging is also a lens on thinking and creativity. Minsky shows in 1967 that poorly understood ideas can be expressed as a program and then corrected on the program; debugging as a stance toward thinking goes back to Papert. ⁴ For Popper, criticism is central: a problem is followed by a tentative theory, then the attempt at error elimination, and from that a new problem usually arises. ⁵ I read this as debugging. A program is a conjecture, a failing test is the refutation.

According to Popper there is no logic for generating ideas, but there is one for testing them. ⁵ That gives a division of labor: operators generate fallible conjectures, and the core is criticism. In Minsky, critics recognize problem types and activate fitting ways of thinking. That is not Popper's criticism, the parallel is my own. ⁴

Criticism has to be independent of the source of the conjecture. A model that criticizes itself in the same conversation is weak for that: without external feedback, self-correction on reasoning tasks mostly improves nothing and sometimes gets worse. ⁶ The reason is probably that generator and critic share the same assumptions; that is my interpretation, not the authors'. Criticism gets stronger step by step: a verifier agent with a fresh context, then an executable check such as a test, a grep or a REPL form, which does not depend on the model at all. That is why my skills demand runs instead of narratives and visibly mark what is unchecked.

The layer analysis is built this way already: each level is a conjecture about the class of errors, the search for siblings is the attempt at refutation, and a level that finds nothing is dropped. Operators themselves are conjectures too. They survive when they bring findings.

• Pre-mortem, assumption stress test, router and neighborhood search are not tested on real cases. Only the layer analysis has produced findings.
• Whether the skill generates the findings on L2 and L3 or only structures them, I do not know.
• That the debugging moves are already in the model is a conjecture. I cannot support it from the inside.
• Whether the layer analysis brings deeper fixes than a bare instruction has not been tested by ablation.

• Which moves show up repeatedly in real debugging sessions, and which of them have no operator yet?
• How do you evaluate an operator? My proposal: it found instances that would have stayed undiscovered without it. Is that enough?
• How many operators do you really need, and how do you recognize that one is superfluous?
• Which axis is still missing? And who decides which axes a problem needs?
• Does the layer analysis really change where the model stops with the fix, or would it have reached the same depth without the skill?

There is no field with a fixed name for this. There is a reason: Reichenbach separated the context of discovery from the context of justification in 1938. [Reichenbach]⁵ Popper had already denied in 1934 that there is a logic of discovery: only testing is open to logical analysis, and invention belongs to psychology for him. [Popper]⁵ The thinking moves migrated into psychology, mathematics education and AI. Here is the trail I have found so far.