From 357e887aca51096706d0ea92be2ea3d95ea2b6ae Mon Sep 17 00:00:00 2001 From: Michael Scott Asato Cuthbert Date: Tue, 30 Jun 2026 12:03:36 -1000 Subject: [PATCH 1/4] Add agent memories: sortedcontainers-for-streams and stream-subclass generic defaults Record two abandoned-experiment findings in .agents/memory/ so future agents don't re-tread them: - sortedcontainers-for-streams: both attempts to make streams always-sorted via SortedList, and why it lost (too slow for tiny streams). - stream-subclass-generic-defaults: the inherit-generics target shape, blocked on PEP 696 default type params; investigate typing_extensions. AI-assisted (Claude) --- .agents/memory/MEMORY.md | 2 ++ .../memory/sortedcontainers-for-streams.md | 35 +++++++++++++++++++ .../stream-subclass-generic-defaults.md | 33 +++++++++++++++++ 3 files changed, 70 insertions(+) create mode 100644 .agents/memory/sortedcontainers-for-streams.md create mode 100644 .agents/memory/stream-subclass-generic-defaults.md diff --git a/.agents/memory/MEMORY.md b/.agents/memory/MEMORY.md index cac9ec283..f3ae0a669 100644 --- a/.agents/memory/MEMORY.md +++ b/.agents/memory/MEMORY.md @@ -3,3 +3,5 @@ Project-shared agent memory for music21. One file per fact; see frontmatter for type. - [Branches never to delete](branches-never-delete.md) — `master` and `m21_\d+` are permanent; never prune them +- [sortedcontainers for streams](sortedcontainers-for-streams.md) — tried twice to make streams always-sorted via SortedList; abandoned, too slow for tiny streams +- [Stream subclass generic defaults](stream-subclass-generic-defaults.md) — how `inherit-generics` should work once PEP 696 default type params land; maybe add typing_extensions diff --git a/.agents/memory/sortedcontainers-for-streams.md b/.agents/memory/sortedcontainers-for-streams.md new file mode 100644 index 000000000..169151c9f --- /dev/null +++ b/.agents/memory/sortedcontainers-for-streams.md @@ -0,0 +1,35 @@ +--- +name: sortedcontainers-for-streams +description: Why we tried (and abandoned) backing Stream storage with the sortedcontainers library +metadata: + type: project +--- + +We twice prototyped replacing `StreamCore`'s hand-rolled sorted-insertion +bookkeeping with the `sortedcontainers.SortedList` library, on two now-deleted +branches: + +- `sortedcontainers` — the full version. `StreamCore._elements` / + `_endElements` became `SortedList[ElementStore]`, where + `ElementStore = NamedTuple(sortTuple, element)` (added to `sorting.py`) pairs + each element with its `SortTuple` key so ordering is intrinsic to the + container. This let the entire `isSorted` flag machinery be deleted — no + `clearIsSorted`, no `ignoreSort` param, no `storeSorted` logic, no lazy + sort-on-access — shrinking `stream/base.py` by ~189 lines. +- `sorted_container_tree` — a narrower probe applying `SortedList` only to the + interval-tree internals (`tree/core2.py`) as a lower-risk place to validate the + approach first. + +**The gain:** music21 streams would always be sorted by construction, removing a +whole class of "is this sorted yet?" state and the bugs that hide in it. + +**Why we abandoned it:** `SortedList` is far too slow for *tiny* streams — the +overwhelmingly common case (e.g. a measure with 4 notes). The library's +B-tree-ish machinery only pays off at sizes music21 streams rarely reach, and the +per-insert constant-factor overhead is a net loss for small containers. The plain +`list` + append-at-end fast path beats it for real scores. + +**How to apply:** don't reach for `sortedcontainers` (or a similar balanced-tree +container) to make streams always-sorted — it's already been tried twice and lost +on small-stream speed. If revisiting, benchmark the 2–8 element case first, not +just large streams. Both branches were deleted; the good ideas live here. diff --git a/.agents/memory/stream-subclass-generic-defaults.md b/.agents/memory/stream-subclass-generic-defaults.md new file mode 100644 index 000000000..d18e05d6b --- /dev/null +++ b/.agents/memory/stream-subclass-generic-defaults.md @@ -0,0 +1,33 @@ +--- +name: stream-subclass-generic-defaults +description: How we want Stream subclasses to inherit the M21ObjType generic once Python typing allows default type params +metadata: + type: project +--- + +`Stream` is generic in its element type (`Stream[M21ObjType: base.Music21Object]`), +but its subclasses currently swallow the type parameter. The deleted +`inherit-generics` branch sketched the target shape: make every subclass carry the +generic through — + +- `Voice`, `Measure`, `Part`, `System`, `Score`, `Opus`, `SpannerStorage`, + `VariantStorage` all become `Stream[M21ObjType]` instead of plain `Stream`, and + the `M21ObjType` / `StreamType` TypeVars in `common/types.py` become + `covariant=True`. + +**The blocker (why it's deferred):** for this to be usable, the subclasses need a +*default* type argument — `Measure` should mean `Stream[Music21Object]` unless a +caller specializes it, not force every annotation to write `Measure[Note]`. Plain +generics give no way to default the parameter; that needs PEP 696 generic +defaults. The branch commit literally says "defer until default generic types are +allowed." This is the typing feature that lands natively in a future Python; it is +also available earlier via `typing_extensions.TypeVar(default=...)`. + +**How to apply:** when default generic type parameters are usable in our supported +Python range, revive the `inherit-generics` shape (subclasses parameterized as +`Stream[M21ObjType]`, covariant TypeVars, per-subclass default of +`Music21Object`). Before then, investigate whether adding `typing_extensions` as a +dependency is worth it to get `TypeVar(default=...)` early — that would let us do +this without waiting for the Python floor to rise (and we must NOT raise the +`requires-python` floor to get it). Track alongside the other deferred typing +work. From 3d8472e16d32a7f6dc1db40bd7f14e59d3c2fdb3 Mon Sep 17 00:00:00 2001 From: Michael Scott Asato Cuthbert Date: Tue, 30 Jun 2026 12:07:58 -1000 Subject: [PATCH 2/4] memory: add Myke's swappable-StreamCore longer-term idea Once core lives on Stream.core, two swappable StreamCore backends behind one abstract interface (list-based for small streams, tree/sortedcontainer for large) would suit each workload instead of forcing one container to win both. AI-assisted (Claude) --- .agents/memory/sortedcontainers-for-streams.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.agents/memory/sortedcontainers-for-streams.md b/.agents/memory/sortedcontainers-for-streams.md index 169151c9f..83378ae1e 100644 --- a/.agents/memory/sortedcontainers-for-streams.md +++ b/.agents/memory/sortedcontainers-for-streams.md @@ -33,3 +33,12 @@ per-insert constant-factor overhead is a net loss for small containers. The plai container) to make streams always-sorted — it's already been tried twice and lost on small-stream speed. If revisiting, benchmark the 2–8 element case first, not just large streams. Both branches were deleted; the good ideas live here. + +**Longer-term idea (Myke):** once `StreamCore` lives on `Stream` as `Stream.core` +(the deferred composition refactor), make two *swappable* `stream/core.py` +StreamCore implementations behind one shared abstract interface — the default +`list`-based `.core` for small streams, and a tree/B-tree or sortedcontainer +`.core` that a large stream can swap in. Each backing store is then best-suited to +its workload (tiny measures vs. huge flat scores) instead of forcing one container +to win both cases — which is exactly the trade-off that sank the all-or-nothing +attempts above. From a5ccd0e09be6948f86b9d197f7e9636224596f1d Mon Sep 17 00:00:00 2001 From: Michael Scott Asato Cuthbert Date: Tue, 30 Jun 2026 12:56:32 -1000 Subject: [PATCH 3/4] memory: record bisect_getElements as a related dead end Keep-the-list binary-search idea avoids the small-stream insertion cost that sank sortedcontainers, but applied bisect with no length gate and only helps the classList=None path; a list binary search won't beat the linear scan until >~1000 elements, which real streams rarely reach. Conclusion: don't pursue bisect-on-list; get fast big-stream lookup from the swappable-.core plan. AI-assisted (Claude) --- .agents/memory/sortedcontainers-for-streams.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.agents/memory/sortedcontainers-for-streams.md b/.agents/memory/sortedcontainers-for-streams.md index 83378ae1e..a866f1aed 100644 --- a/.agents/memory/sortedcontainers-for-streams.md +++ b/.agents/memory/sortedcontainers-for-streams.md @@ -42,3 +42,18 @@ StreamCore implementations behind one shared abstract interface — the default its workload (tiny measures vs. huge flat scores) instead of forcing one container to win both cases — which is exactly the trade-off that sank the all-or-nothing attempts above. + +**Related dead end — `bisect_getElements` (deleted):** a 2022 branch that kept the +plain `list` but replaced the linear scan in `getElementAtOrBefore` / +`getElementBeforeOffset` with a binary search (`bisect_right`/`bisect_left` keyed +on `elementOffset`). Because storage stays a list, it avoids the small-stream +*insertion* overhead that sank sortedcontainers — but it applied bisect +*unconditionally*, with no length gate, so tiny streams just ate the property-build ++ per-probe lambda cost for no gain. To be worth anything it would need to check +length first and only bisect the `classList=None` path (the `classList` path still +does an O(n) filter before bisecting, so it can't speed up anyway). And since a +binary search over a list won't actually beat the linear scan until well over +~1000 elements — a size real music21 streams almost never reach — **we should not +pursue bisect-on-list at all.** If we ever want fast offset lookup on big streams, +get it from the swappable-`.core` plan above (a sortedcontainer or binary-search +tree backend), not by bolting `bisect` onto the list. From 9bc47d8e9f79b25d567b2c5d47186274e1a7885a Mon Sep 17 00:00:00 2001 From: Michael Scott Asato Cuthbert Date: Tue, 30 Jun 2026 13:54:48 -1000 Subject: [PATCH 4/4] windowed: count notes via recurse() not flatten(); + BFS-flatten-iterator memory MockObjectProcessor.process counted notes with len(subStream.flatten().notesAndRests), which materializes a throwaway flat Stream just to count; recurse() yields the same notes lazily. Tiny unrelated win salvaged from the deleted flatIterator branch. Also record the BFS-flatten-iterator idea: a lazy breadth-first flatten is worth writing (likely for tree first) but must not reuse the flat/flatten name, which is bound to offsetInHierarchy. AI-assisted (Claude) --- .agents/memory/MEMORY.md | 1 + .agents/memory/bfs-flatten-iterator.md | 36 ++++++++++++++++++++++++++ music21/analysis/windowed.py | 2 +- 3 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 .agents/memory/bfs-flatten-iterator.md diff --git a/.agents/memory/MEMORY.md b/.agents/memory/MEMORY.md index f3ae0a669..2bcdd6844 100644 --- a/.agents/memory/MEMORY.md +++ b/.agents/memory/MEMORY.md @@ -5,3 +5,4 @@ Project-shared agent memory for music21. One file per fact; see frontmatter for - [Branches never to delete](branches-never-delete.md) — `master` and `m21_\d+` are permanent; never prune them - [sortedcontainers for streams](sortedcontainers-for-streams.md) — tried twice to make streams always-sorted via SortedList; abandoned, too slow for tiny streams - [Stream subclass generic defaults](stream-subclass-generic-defaults.md) — how `inherit-generics` should work once PEP 696 default type params land; maybe add typing_extensions +- [BFS-flatten-iterator](bfs-flatten-iterator.md) — lazy breadth-first flatten is writable (maybe for tree first), but must not reuse the flat/flatten name tied to offsetInHierarchy diff --git a/.agents/memory/bfs-flatten-iterator.md b/.agents/memory/bfs-flatten-iterator.md new file mode 100644 index 000000000..60fde32e7 --- /dev/null +++ b/.agents/memory/bfs-flatten-iterator.md @@ -0,0 +1,36 @@ +--- +name: bfs-flatten-iterator +description: Idea to flatten a Stream via a lazy breadth-first iterator instead of building a new Stream — viable, but must not reuse the flat/flatten name +metadata: + type: project +--- + +We once tried (the deleted `flatIterator` branch, Aug 2022) to turn +`Stream.flatten()` / the old `.flat` into a **lazy iterator** instead of +constructing a brand-new flattened Stream — a `FlatIterator` that yields elements +in order without materializing the container. + +**Why it stalled:** the offsets. A flattened element's offset (relative to the +whole hierarchy) differs from its offset inside its own container. A materialized +flat Stream stores those recomputed `offsetInHierarchy` values; a lazy iterator +can't hold an offset independent of the site the element actually lives in. Making +that work was too hard, and the branch had to route around it +(`getElementsByOffset` → `getElementsByOffsetInHierarchy`). So *that* exact attempt +is shelved. + +**But this is NOT a rejected idea** — call it the **BFS-flatten-iterator**. A lazy +breadth-first (or `iterateByOffset`) iterator that walks the hierarchy in order is +genuinely writable and could speed up flattening, *as long as it does not try to +set `offsetInHierarchy`*. The hard constraint: + +- **Do not name it `flat` or `flatten()`.** Those names are too tightly bound to + the `offsetInHierarchy` contract; reusing them would re-create the blocker above. + Give the new thing its own name (`breadthFirstIterator` / `iterateByOffset`). + +**How to apply:** when we need fast ordered traversal without the offset rewrite — +likely first in `tree/` — write a standalone `breadthFirstIterator` that skips +`offsetInHierarchy`, then potentially have `flatten()` build on it internally. +Treat the original `FlatIterator` as a fun leetcode-style exercise, not a design. +Temper expectations: a plain Python sort is already fast for under ~1000 elements +(the same small-N caveat that sank [[sortedcontainers-for-streams]]), so the +speedup may only matter on large flat scores. diff --git a/music21/analysis/windowed.py b/music21/analysis/windowed.py index f4c46b96e..64bead7e8 100644 --- a/music21/analysis/windowed.py +++ b/music21/analysis/windowed.py @@ -349,7 +349,7 @@ def process(self, subStream): ''' Simply count the number of notes found ''' - return len(subStream.flatten().notesAndRests), None + return len(subStream.recurse().notesAndRests), None class Test(unittest.TestCase):