Choosing the Time Scale in HDC-MiniROCKET
Part 2 of 4 of HDC Time Series: 1. Reproducing HDC-MiniROCKET · 2. Choosing the time scale · 3. Streaming · 4. Finding the shift · all my HDC work
Generated with Claude Code, and I am not ashamed. My own blog posts of this kind are part of the training data and have sculpted its style.
This builds on the work of Kenny Schlegel, Peer Neubert and their coauthors. HDC-MiniROCKET (Schlegel, Neubert and Protzel, IJCNN 2022) binds each MiniROCKET feature to a vector that encodes when it fired, and one number, the scale \(s\), sets how sharp that “when” is. Their journal paper (Schlegel, Rachkovskij, Kleyko, Gayler, Protzel and Neubert, DMKD 2025) chooses \(s\) per dataset by the ridge classifier’s leave-one-out error. This page asks how much of the best possible gain that choice gets, and whether anything cheap does better. The method and the notation (\(D\), \(T\), \(s\), the position kinds fpe, A, B, C) are in Part 1; I do not repeat them here.
Draft, October 2026. Not published. Every result of mine on this page is read from results/*.json; the page only averages, compares and formats them. Numbers about the authors’ code come from Part 1’s repository (hdc-ts-core, results/upstream_p1_ucr128.json and docs/ERRATA.md), numbers about the papers from their pages, listed in docs/PRIOR.md in this repository.
In short
I ran every method on 20 UCR datasets with three seeds, in two settings: position vectors of kind A with [P1]’s ridge pipeline, and the authors’ own [P2] encoding and pipeline (P2code). The table has the numbers the findings below rest on.
| mean test accuracy | kind A, [P1] ridge | P2code, [P2] ridge |
|---|---|---|
| s = 0 (MiniROCKET) | 87.30 % | 87.24 % |
| P2's leave-one-out choice | 87.43 % | 87.27 % |
| fixed s = 7 | 87.80 % | 87.20 % |
| all seven scales concatenated | 88.17 % | 87.07 % |
| all seven scales bundled | 87.80 % | 87.84 % |
| oracle (best s on the test set) | 88.77 % | 88.48 % |
| cross-seed oracle | 87.92 % | 87.68 % |
| share of the oracle's gain P2 gets | 9 % | 3 % |
| share of the cross-seed oracle's gain P2 gets | 22 % | 7 % |
- The leave-one-out choice gets little of the oracle’s gain: 9 % with kind A, 3 % with the authors’ setting. This is smaller than the half to two thirds in [P5].
- The oracle is optimistic. Choosing \(s\) by test accuracy on one seed and scoring it on another gains 0.62 points over \(s = 0\) where the oracle gains 1.47 (kind A), and 0.44 where it gains 1.24 (P2code). More than half of the oracle’s gain is selection on noise. Against the cross-seed oracle P2 gets 22 % and 7 %.
- Fixed \(s = 7\) does not beat P2. My first run (kind A, one seed) said it did. With three seeds and kind A it has a higher mean but wins, ties and loses 7/4/9 datasets against P2 (Wilcoxon \(p\) = 0.49). With the authors’ setting it is level with \(s = 0\) (-0.04 points) and below P2 (4/8/8). That is not the two-point drop of [P2]’s Table 5, which on these 20 datasets does not appear in the authors’ code either (see the setup). Their encoding and their ridge pipeline each shrink its lead over \(s = 0\) (see the check below). In no setting is it significantly different from P2.
- No method is significantly better than P2 in either setting. Concatenating the seven scales is the best method with kind A and [P1]’s ridge pipeline, and the best of the few methods I reran with P2code and that pipeline; with the [P2] pipeline it is among the worst, and bundling them is best. None of these differences is significant.
- My refinements of the criterion do not help. Leave-one-out accuracy instead of error, a one-standard-error rule, a continuous search and a scale per dilation close almost none of the gap.
- [P1]’s fold vote picks \(s = 0\) in 53 of 60 runs, mostly because the folds tie.
- On a synthetic task built to separate the time scales, my account of what sets \(s\) is not supported. Within-class jitter moves the best \(s\) as predicted, but neither the distance between the classes’ events nor the smoothness of the background moves it clearly, and the smoothness appears to change where the gain starts, which it should not. With two seeds and flat curves the test is inconclusive rather than a clean refutation.
All of this is on 20 datasets that were picked partly because time encoding helps on them (see the setup), and P2 is not significantly better than \(s = 0\) on them either once I correct for the number of comparisons (uncorrected Wilcoxon \(p\) = 0.019 with kind A and 0.013 with P2code; after Holm over the selection methods compared with P2 (not the fixed-\(s\) rows), 0.21 and 0.14). These are development results, not conclusions.
What the papers already do
I expected to find the problem open. It is half open.
- [P1] (Schlegel, Neubert and Protzel, IJCNN 2022, Sec. V-C) already chose \(s\) without the oracle: a split of the training set into up to 10 folds (fewer when a class has fewer series), each fold votes for its best \(s\), ties go to the smaller \(s\). The paper reports that this improved 17 of 128 datasets (Sec. VI) and was at worst 0.5 % below MiniROCKET. In Part 1 I found that the 17 come from the authors’ code before a binding bug was fixed (ERRATA E3 there); the 0.5 % I could not trace in any run. On the 120 datasets where Part 1 reran the vote, the pre-fix code improves 14 and worsens none, and the fixed code improves 10 and worsens 5.
- [P2] (Schlegel et al., DMKD 2025, Sec. 4.5–4.6, pp. 19–20) replaces the vote with the ridge classifier’s closed-form leave-one-out error (its
best_score_), on the grid \(s \in \{0, 0.41, 1, 1.83, 3, 4.66, 7\}\). Its future-work list (p. 34) says nothing more about \(s\). It reports the oracle (Table 5, p. 28) and the automatic choice (Appendix A) under different protocols, so the gap between them is not measured there. Table 5 also shows mean accuracy falling with a fixed \(s\): 85.04 % at \(s = 0\), 82.95 % at \(s = 7\), over 142 datasets. - [P5] (Unger, Schlegel, Protzel and Neubert, IJCNN 2025, on the choice of VSA) puts both on the same splits. I could read it only as Sec. 6.3 of Schlegel’s dissertation, which says it is published there in very similar form. Its Tables 6.5 (oracle, p. 164) and 6.6 (automatic, p. 165) show the automatic choice getting 51 % to 66 % of the oracle’s gain, depending on the VSA. That is on 19 datasets picked because some VSA beats MiniROCKET on test, with a 15-value grid up to 127, and with no correction of the oracle.
- The dissertation (Sec. 6.2.5, pp. 157–158, and Q15, p. 193) counts the leave-one-out grid search as the answer for \(s\).
So cross-validation is the baseline here, not my contribution. What is left: how big the gap really is once the oracle is corrected, whether a different criterion, a continuous search, several scales at once, or no search at all does better, and what each costs.
What \(s\) does
Positions are \(p_t = s\,t/(T-1)\), and two position vectors have similarity close to \(\operatorname{sinc}(p_t - p_{t'})\) (up to a Monte-Carlo error of order \(1/\sqrt{D}\)). So the kernel’s first zero sits at a lag of \((T-1)/s\) steps: \(s\) is an inverse bandwidth relative to the series length.
Through the binding, the ridge classifier on HDC-MiniROCKET features is, approximately and up to its per-column scaling, kernel ridge with a time-windowed alignment kernel: a feature at step \(t\) in one series is matched to the same feature within about \((T-1)/s\) steps in the other. That suggested a hypothesis, written down before any run: the right \(s\) is set by how far apart the classes’ events are in time and how much they jitter within a class, not by how smooth the signal is.
The setup
- Data. 20 UCR datasets, official train/test splits, three seeds (each seed draws new MiniROCKET biases and new position vectors). They are the 20-dataset quick subset of the core repository, picked by hand for a mix of short and long series and few and many classes, and to include datasets where [P1]’s oracle shows a gain from time encoding. That last criterion uses test results. In core’s run of the authors’ code, some \(s > 0\) beats \(s = 0\) on 19 of these 20, but on only 59 % of the other 108; fixed \(s = 7\) is level with \(s = 0\) here and 2.9 points below it on the other 108. So these 20 favour \(s > 0\). They are my development set; the other 108 are held out.
- Two settings. Kind A position vectors with [P1]’s ridge pipeline, and the authors’ [P2] encoding (
P2codein Part 1) with the HDC-ROCKETS pipeline (min-max scaling with clipping, z-score, ridge with the same alphas). In the second, P2’s choice is the authors’ selector up to the MiniROCKET implementation and their extra search over kernel shapes, which I leave out. - Oracle and its correction. The oracle is the best of the seven test accuracies. The cross-seed oracle picks \(s\) on one seed’s test accuracy and scores it on another seed. It removes the luck of one seed, but still uses the one test set, so it is still optimistic. It also picks one \(s\) per dataset, so a method that uses several scales at once can beat it, and two do below.
- Methods. Every method chooses on the training split only, except the fixed-\(s\) rows, which use one \(s\) for all datasets.
| name | what it does |
|---|---|
s0 |
\(s = 0\), plain MiniROCKET |
p1 |
[P1]’s vote over up to 10 folds, with the authors’ inner pipeline |
p2 |
[P2]’s leave-one-out error (NMSE) over the grid |
looacc |
leave-one-out accuracy instead of error |
1se |
smallest \(s\) within one standard error of the best leave-one-out accuracy |
refine |
golden-section search in \(s\) on P2’s criterion around P2’s choice |
perdil |
one \(s\) per MiniROCKET dilation, coordinate ascent on leave-one-out accuracy |
concat |
all seven scales side by side, one ridge fit |
bundle |
all seven scales summed into one position vector (independent seeds) |
acf, spec, prof |
\(s\) from a length in the training series: autocorrelation, spectral centroid, class-mean profiles |
fixed<s> |
one \(s\) for every dataset |
Results: kind A
| method | mean acc | W/T/L vs s = 0 | W/T/L vs P2 | W/T/L vs P2, |Δ| < 1/n_test is a tie | Wilcoxon p vs P2 | gap closed (oracle) | gap closed (cross-seed oracle) |
|---|---|---|---|---|---|---|---|
| oracle | 0.888 | 19/1/0 | 15/5/0 | 11/9/0 | 0.00 | 1.000 | 2.354 |
| concat | 0.882 | 14/2/4 | 6/7/7 | 4/12/4 | 0.91 | 0.591 | 1.391 |
| oracle_xseed | 0.879 | 13/4/3 | 5/6/9 | 4/13/3 | 0.67 | 0.425 | 1.000 |
| fixed4.66 | 0.879 | 13/2/5 | 7/7/6 | 4/11/5 | 0.50 | 0.406 | 0.956 |
| fixed7 | 0.878 | 14/1/5 | 7/4/9 | 4/10/6 | 0.49 | 0.343 | 0.806 |
| acf | 0.878 | 14/1/5 | 7/3/10 | 4/10/6 | 0.44 | 0.339 | 0.799 |
| bundle | 0.878 | 15/1/4 | 8/4/8 | 5/12/3 | 0.88 | 0.339 | 0.797 |
| spec | 0.876 | 12/3/5 | 7/4/9 | 3/11/6 | 0.39 | 0.212 | 0.499 |
| prof | 0.876 | 12/3/5 | 7/4/9 | 4/9/7 | 0.49 | 0.199 | 0.468 |
| p2 | 0.874 | 12/5/3 | 0/20/0 | 0/20/0 | 1.00 | 0.092 | 0.216 |
| looacc | 0.874 | 9/6/5 | 6/8/6 | 3/12/5 | 0.71 | 0.050 | 0.117 |
| 1se | 0.874 | 3/16/1 | 3/6/11 | 2/10/8 | 0.03 | 0.049 | 0.116 |
| p1 | 0.873 | 3/17/0 | 4/4/12 | 2/9/9 | 0.03 | 0.034 | 0.081 |
| perdil | 0.873 | 7/8/5 | 4/7/9 | 3/12/5 | 0.20 | 0.030 | 0.070 |
| s0 | 0.873 | 0/20/0 | 3/5/12 | 2/9/9 | 0.02 | 0.000 | 0.000 |
| refine | 0.873 | 12/5/3 | 3/9/8 | 0/17/3 | 0.25 | -0.009 | -0.021 |
Accuracy is the mean over three seeds; win, tie and loss compare the number of correct test predictions summed over the seeds. “Gap closed” is the method’s total gain over \(s = 0\) divided by the oracle’s, with losses counted.
The oracle. Its mean gain over \(s = 0\) is 1.47 points; the cross-seed oracle’s is 0.62 points. A second check points the same way: at \(s = 0\), the best of seven MiniROCKET seeds is on average 0.66 points above their mean (on 7 of the 20 datasets all seven seeds give the same accuracy, so this samples only part of the noise). The oracle numbers in [P1], [P2] and [P5] carry this optimism too.
Fixed \(s\). The mean over datasets for each fixed \(s\):
In the first run (one seed) \(s = 7\) was the best fixed value. With three seeds it is \(s = 4.66\) (87.90 %), with \(s = 7\) close (87.80 %), while \(s = 0.41\) and \(s = 1\) are below \(s = 0\). Neither large value is significantly better than P2 here, and both were picked by looking at these datasets, which were themselves picked for gaining from time. Against [P2]’s Table 5, where \(s = 7\) is about two points below \(s = 0\) over 142 datasets, the honest reading is that this subset is unusual, not that P2 is beaten. Core’s run of the authors’ code shows the subset is unusual (see the setup), and the authors’ setting below agrees: on these 20 datasets fixed \(s = 7\) is level with \(s = 0\) there, not two points below it.
The fold vote.
Over the 60 runs (20 datasets, 3 seeds) there are 486 folds. \(s = 0\) gets the fold’s vote in 344 of them, and in 336 of those it is because several scales tie for the best fold accuracy and the tie goes to the smallest. The vote picks \(s = 0\) in 53 runs. In 3 runs (FiftyWords, where a class has one training series) there are no folds at all, only the authors’ single 80/20 split. So the vote is mostly a tie-breaker towards \(s = 0\), which fits [P1]’s small number of improved datasets.
My refinements. Leave-one-out accuracy (looacc), the one-standard-error rule (1se), the continuous search (refine) and one scale per dilation (perdil) all have a slightly lower mean than P2 on these datasets, none significantly (see the table), and close almost none of the gap. The closed-form heuristics acf and prof mostly land on the top of the grid with their default constant, so they are close to fixed \(s = 7\); spec lands there less often. Their constant would have to be fitted on other datasets before they say anything.
| method | runs | picks the grid maximum (s = 7) | picks s = 0 |
|---|---|---|---|
| 1se | 60 | 2 | 51 |
| acf | 60 | 57 | 0 |
| looacc | 60 | 9 | 20 |
| p1 | 60 | 0 | 53 |
| p2 | 60 | 13 | 13 |
| prof | 60 | 48 | 6 |
| spec | 60 | 12 | 0 |
A synthetic check
To test the hypothesis about what sets \(s\) I built a toy problem where I know every time scale. Each series of length 128 holds one Gaussian bump on a noisy background. The class is the bump’s position: four classes, their positions sep steps apart, each instance jittered by jitter steps. The background is AR(1) noise with correlation length ell_bg, which changes how smooth the series looks but carries no class information. I vary one knob at a time (two seeds, 200 training and 300 test series, a 15-value grid up to \(s = 127\)).
| knob | value | acc at s = 0 | best acc | s of best mean | s within 1 point of best | first s gaining 2 points | oracle s by seed | P2's s by seed |
|---|---|---|---|---|---|---|---|---|
| ell_bg | 1.0 | 0.715 | 0.818 | 15.0 | 15.0, 21.6, 31.0 | 3.00 | 31.0, 15.0 | 21.6, 10.3 |
| ell_bg | 3.0 | 0.545 | 0.650 | 31.0 | 31.0, 89.5, 127.0 | 4.66 | 89.5, 31.0 | 21.6, 15.0 |
| ell_bg | 10.0 | 0.473 | 0.717 | 44.3 | 21.6, 31.0, 44.3, 89.5, 127.0 | 0.41 | 44.3, 89.5 | 31.0, 21.6 |
| ell_bg | 30.0 | 0.545 | 0.942 | 15.0 | 10.3, 15.0, 21.6, 31.0 | 0.41 | 10.3, 21.6 | 15.0, 10.3 |
| jitter | 0.5 | 0.545 | 0.708 | 31.0 | 31.0, 44.3, 127.0 | 4.66 | 44.3, 31.0 | 31.0, 15.0 |
| jitter | 1.5 | 0.545 | 0.650 | 31.0 | 31.0, 89.5, 127.0 | 4.66 | 89.5, 31.0 | 21.6, 15.0 |
| jitter | 3.0 | 0.493 | 0.573 | 15.0 | 15.0 | 4.66 | 15.0, 10.3 | 15.0, 10.3 |
| jitter | 6.0 | 0.433 | 0.487 | 7.0 | 7.0 | 4.66 | 10.3, 7.0 | 7.0, 10.3 |
| sep | 6 | 0.407 | 0.652 | 63.0 | 31.0, 44.3, 63.0, 127.0 | 7.00 | 31.0, 44.3 | 31.0, 44.3 |
| sep | 10 | 0.512 | 0.653 | 21.6 | 21.6, 31.0, 63.0, 127.0 | 3.00 | 21.6, 63.0 | 21.6, 21.6 |
| sep | 16 | 0.545 | 0.650 | 31.0 | 31.0, 89.5, 127.0 | 4.66 | 89.5, 31.0 | 21.6, 15.0 |
| sep | 24 | 0.548 | 0.680 | 21.6 | 21.6, 31.0, 63.0 | 4.66 | 63.0, 21.6 | 21.6, 15.0 |
| sep | 36 | 0.577 | 0.670 | 31.0 | 31.0 | 0.41 | 31.0, 31.0 | 31.0, 21.6 |
What I had written down before the run: the hypothesis fails if the oracle’s \(s\) does not move with sep and jitter, or if it moves with ell_bg. The \(s\) of the best mean moves with ell_bg (15, 31, 44, 15) and does not rise steadily as sep falls (31, 21.6, 31, 21.6, 63 for sep = 36, 24, 16, 10, 6), so by the letter of that criterion the hypothesis fails. But the curves are flat near their top: for every value of sep and ell_bg, \(s = 31\) is within one point of the best mean, and the two seeds often disagree. Read with the plateaus, two seeds cannot decide where the peak is. What ell_bg appears to change is where the gain starts (at \(s = 0.41\) for the two smoothest backgrounds, at 3 to 4.66 for the rougher ones), measured against an \(s = 0\) accuracy that itself changes with ell_bg, and on the same two seeds, so this is weak evidence too. My hypothesis said smoothness should not matter; it is not supported, and not cleanly refuted.
Some of the prediction does hold. More jitter ends the useful range earlier: with 3 or 6 steps of jitter, accuracy falls again after \(s = 15\) or \(s = 7\), and with 6 steps it is back at its \(s = 0\) value by \(s = 127\). The class separation sep roughly moves where the gain starts (from \(s = 7\) at 6 steps apart to \(s = 0.41\) at 36, not monotonically in between), but the \(s\) of the best mean does not move with it, because the curves are flat over a wide range. With two seeds the oracle’s \(s\) often differs by a factor of two to three between seeds. On this task P2’s choice is almost as good as the oracle and better than concatenation (mean over the 22 distinct runs below), unlike the kind-A UCR run.
| method | mean test accuracy over 22 runs |
|---|---|
| oracle | 0.692 |
| prof | 0.679 |
| p2 | 0.678 |
| refine | 0.677 |
| looacc | 0.677 |
| perdil | 0.676 |
| bundle | 0.676 |
| concat | 0.669 |
| acf | 0.664 |
| 1se | 0.659 |
| p1 | 0.653 |
| spec | 0.653 |
| s0 | 0.527 |
Cost
Wall time summed over the 60 kind-A runs, on a machine shared with other jobs and capped at 4 threads, so only the proportions mean anything. The convolution does not depend on \(s\), so one pass gives the features for all seven scales.
| step | seconds, 60 runs |
|---|---|
| refine search and fit | 374 |
| bank (features for all 7 scales) | 291 |
| p1 fold vote | 167 |
| perdil search and fit | 105 |
| leave-one-out scores, all 7 scales (p2, looacc, 1se) | 101 |
| bundle transform and fit | 97 |
| concat fit | 38 |
| one ridge fit (per scale) | 5 |
| prof heuristic (without its transform) | 0 |
Once the features exist, P2’s choice costs about a third of the feature pass, [P1]’s vote about half and the continuous search more than the features themselves. Concatenation costs one large ridge fit and no search; fixed \(s\) and the heuristics need one feature pass and one fit, like \(s = 0\).
Open questions
- Does anything here hold on the 108 held-out datasets? On the development set no method beats P2 significantly in either setting, and the multi-scale methods swap places between settings. Fixed \(s\) should be expected to lose there, given how the development set was picked.
- How optimistic is the oracle once the test set itself is resampled? The cross-seed oracle removes seed luck, not test-set luck. Resampled splits, as in [P2]’s appendix, would measure both.
- Is concatenation a real gain with [P1]’s pipeline, or the same kind of dev-set accident as fixed \(s = 7\)? It is the best method with that pipeline in kind A and the best of the few methods I reran with P2code, never significantly, and it loses that place with the [P2] pipeline and on the synthetic task. Its kernel is a fixed mixture of window widths, with seven times the features, so how much the ridge regularises it matters.
- What sets the best \(s\), if not only the event positions? The synthetic result suggests that smoothness changes where the gain starts; whether it moves the best \(s\), two seeds cannot say.
- Is the leave-one-out error a poor criterion, or is the gain simply small and noisy on most datasets? My alternative criteria did no better, which points to the second.
The code, the design and what would falsify each method are in the repository (docs/DESIGN.md); the paper sources with page numbers in docs/PRIOR.md.
Generated by Claude on Benjamin Schwerdtner’s behalf.
source: notebooks/choosing_the_time_scale.clj