Adds a workspace indexer plugin backed by @llamaindex/semtools, a Rust
CLI that does semantic search via model2vec. The plugin auto-installs
semtools on first use and delegates indexing to semtools' lazy approach
(embed-on-search). Includes 23 tests covering subclass contract, plugin
discovery, factory integration, CLI invocation, result parsing, filtering,
error handling, and edge cases.
- Remove `from __future__ import annotations` from all new workspace files
- Convert TYPE_CHECKING imports to real imports in base.py (no circular deps)
- Quote self-referential forward ref in config.py model_validator
- Add null checks on spec.loader in plugin discovery to satisfy ty
Delete the old workspace/indexer.py and workspace/search.py modules.
All test imports now use workspace.default.DefaultIndexer directly.
Backwards-compat re-exports removed from workspace/__init__.py.
- Narrow markdown pipeline result with isinstance assert (static type
correctness; .code/.tables/.images access no longer leaks Document
abstraction).
- Simplify _execute_with_lock_retry to 5 linear-backoff attempts; the
helper was tuned for WAL schema bootstrap, not repeated retry.
- Add Literal['markdown','code','plain'] alias for pipeline keys so
typos at the dispatch site become type errors.
- Fix misleading stage="discover" label on post-discovery errors;
relabel as "read" where it actually applies.
- Extract _make_config / _write into tests/workspace/conftest.py
fixtures so the two test files share one source.
- Factor str(Path(raw).resolve()) into workspace.constants.resolve_path_prefix
and call from both search_workspace and the CLI command.
- Drop a stale WHAT-comment on the retry backoff line.
Follow-up to the chonkie Pipeline migration. Parallel black-box verification
surfaced four bugs and three minor follow-ups:
- Add PRAGMA busy_timeout via sqlite3.connect(timeout=5.0) — fixes
concurrent index crashes exposed when _build_pipelines removed the
lazy-init skew that previously hid the race.
- Resolve path_prefix in search_workspace (Python API entry) so it
matches the indexer's resolved stored paths, mirroring what the CLI
already does in commands.py.
- Hardcode .hermesignore exclusion in discovery, and add it to
DEFAULT_IGNORE_PATTERNS belt-and-suspenders.
- Extend DiscoveryResult with filtered_count and roll it into
files_skipped so empty/oversized files stop vanishing from summaries.
- Tighten pipelines dict type from dict[str, Any] to dict[str, Pipeline].
- Drop dead {"language": lang} branch in _process_code — CodeChunker
language="auto" never populates the attribute.
- Tighten test_small_markdown_file_is_split_into_modalities to assert
prose doesn't swallow the code fence.
Adds four regression tests covering each fix.
Code-review follow-up to the chonkie Pipeline migration:
- Delete _extract_first_heading and _kind_from_suffix (only callers
were _single_chunk, which was removed in the migration).
- Drop unused `import pytest` in test_indexer_pipeline.py (all xfail
markers removed post-migration).
- Update test_indexer_pipeline.py docstrings to describe post-migration
invariants rather than "current impl" behavior that no longer exists.
Replaces the manual MarkdownChef → chunker → OverlapRefinery wiring (and
the _ChunkerCache / _apply_overlap / _group_overlap_runs / _neural_enforce_size
layer) with three pre-built chonkie.Pipeline instances dispatched by file suffix.
- Drop 'semantic' and 'neural' chunking strategies and their model pins;
RecursiveChunker is the only prose chunker. BM25/FTS5 doesn't benefit
from topical coherence.
- Drop ChunkingConfig.strategy and ChunkingConfig.threshold. No legacy
compatibility — old keys in user configs are silently ignored.
- Drop block_index / src / link / row_count / column_count metadata as not
load-bearing for keyword search.
- Drop _single_chunk short-circuit so small files still flow through the
Pipeline (preserves prose/code split for tiny multimodal markdown).
- Drop manual _apply_overlap; OverlapRefinery.refine_document handles prose
chunks inside the Pipeline, populating Chunk.context directly.
- Bump CHUNKING_PLAN_VERSION from v1 to v2 so existing indexes get
re-built cleanly on upgrade.
Pins the target behavior for the Chonkie Pipeline migration:
- Markdown files produce one ChunkRecord per modality with clean metadata
- Small markdown files remain multimodal (no _single_chunk short-circuit)
- Overlap context propagates and is a suffix of prior chunk's content
- Deprecated strategy/threshold keys load without error
- Signature change forces re-indexing
Marked xfail(strict=True) until the migration lands in the next commit.
alt-glitch