Skip to content

Struct attrributes#268

Merged
JoOkuma merged 35 commits into
royerlab:mainfrom
yfukai:struct_attr
Jun 14, 2026
Merged

Struct attrributes#268
JoOkuma merged 35 commits into
royerlab:mainfrom
yfukai:struct_attr

Conversation

@yfukai

@yfukai yfukai commented Feb 20, 2026

Copy link
Copy Markdown
Contributor

Partially solves #194 by allowing pl.Struct saved and searched in graphs.
Assumes #262 merged.

@codecov-commenter

codecov-commenter commented Feb 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.90955% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.56%. Comparing base (13f9ba8) to head (3759d9c).

Files with missing lines Patch % Lines
src/tracksdata/attrs.py 74.41% 10 Missing and 1 partial ⚠️
src/tracksdata/graph/_sql_graph.py 90.83% 6 Missing and 5 partials ⚠️
src/tracksdata/graph/_rustworkx_graph.py 41.17% 8 Missing and 2 partials ⚠️
src/tracksdata/utils/_dtypes.py 78.94% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #268      +/-   ##
==========================================
- Coverage   87.81%   87.56%   -0.25%     
==========================================
  Files          57       57              
  Lines        4998     5155     +157     
  Branches      877      908      +31     
==========================================
+ Hits         4389     4514     +125     
- Misses        384      409      +25     
- Partials      225      232       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yfukai yfukai marked this pull request as ready for review February 26, 2026 02:51
@yfukai yfukai mentioned this pull request Feb 26, 2026
16 tasks
@yfukai yfukai marked this pull request as draft February 26, 2026 03:14
@yfukai

yfukai commented Feb 26, 2026

Copy link
Copy Markdown
Contributor Author

Maybe I should save the data field-wise in SQLGraph. Let me experiment in this way.

@JoOkuma

JoOkuma commented Feb 27, 2026

Copy link
Copy Markdown
Member

@yfukai, let me know when this is ready for review.

@yfukai

yfukai commented Mar 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, it might take a while, but I'll continue working on this.

@yfukai yfukai marked this pull request as ready for review April 14, 2026 08:21
@yfukai

yfukai commented Apr 14, 2026

Copy link
Copy Markdown
Contributor Author

@JoOkuma I believe this is good to go! Sorry for the large PR.

@JoOkuma

JoOkuma commented Apr 21, 2026

Copy link
Copy Markdown
Member

@yfukai, I need more time to take a look at your PRs. I'll try to get them done before the end of the week.

@yfukai

yfukai commented May 28, 2026

Copy link
Copy Markdown
Contributor Author

Added additional comments! It would be great if you could review this, since this changes the database structure for a "struct"-type attributes @JoOkuma

@JoOkuma JoOkuma left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @yfukai, I haven't finished reviewing yet, but here are some comments.

I'm worried about the codebase's complexity. I'm taking my time to review this.

Comment thread src/tracksdata/attrs.py Outdated
Comment on lines +225 to +226
if isinstance(out, Attr):
out._append_field_path(name)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this if always be true according to the return value?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think it's guaranteed to be Attr before this block. Thanks!

Comment thread src/tracksdata/utils/_dtypes.py
@yfukai

yfukai commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Sure, thanks @JoOkuma! Let me also try to reduce the complexity.

yfukai and others added 8 commits June 4, 2026 01:29
The previous commit (db35287) mistakenly deleted upstream/main's _SQLIDSet
scratch-table machinery, _create_id_scratch_table, the out_degree/copy
bound-variable handling, and the three scratch-table tests — they were diffed
against a stale fork main and wrongly treated as PR-added code. Restore them
verbatim from main; the struct-attr column-expansion simplification from the
previous commit is kept.
@JoOkuma

JoOkuma commented Jun 14, 2026

Copy link
Copy Markdown
Member

Hi @yfukai, sorry for the delayed review of this PR.
I really like the new version; it's much easier to understand than the original version.

There was a very complex merge with the base after the other changes.
I think I solved the conflicts, if there were any mistakes, we can address them in a separate PR, so we don't delay further merging this PR.

#304 This PR is extremely interesting, and I really like the approach.

@JoOkuma JoOkuma merged commit c373252 into royerlab:main Jun 14, 2026
11 of 13 checks passed
@yfukai

yfukai commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Hi @JoOkuma, thanks a lot! I believe this was the largest update before the release. I'll try to finalize #304 quickly.

yfukai added a commit to yfukai/tracksdata that referenced this pull request Jun 17, 2026
Resolve conflicts from upstream's struct-attribute PR (royerlab#268) and the
single->bulk add_node/add_edge refactor:

- attrs.py: take upstream's `columns` property and `f.to_attr().expr`
  filter reduction (handles compound AttrFilters as well as struct-field
  AttrComparisons).
- _rustworkx_graph.py: drop the now-duplicate nested `_extract_field_path`;
  use upstream's module-level `_eval_filter` (supports AND/OR/XOR/NOT).
- _sql_graph.py: adopt upstream's `_to_sql_clause` and the removal of the
  single `add_node`/`add_edge` methods (BaseGraph now delegates to the bulk
  variants). Mask struct flattening is preserved via `_flatten_attrs_for_write`
  on the bulk write paths.

Mask remains stored as a struct attribute (this branch's feature); upstream
left Mask as pl.Object.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
yfukai added a commit to yfukai/tracksdata that referenced this pull request Jun 17, 2026
- Remove duplicate `_StructNamespace` class in attrs.py left by the upstream
  merge (this branch and upstream PR royerlab#268 each added an identical copy).
- Drop the now-empty `if TYPE_CHECKING: pass` block and unused `TYPE_CHECKING`
  import in array/_graph_array.py.
- Fix `_restore_pickled_column_types` docstring to reference the helper it
  actually uses (`_raw_binary_physical_columns`).
- Align Mask.bbox_struct_fields/struct_dtype docstrings with the supported
  1-to-3 dimension range.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants