radiens_drive_catalog

radiens-drive-catalog

Programmatic catalog and sync tool for xdat neural recordings stored on Google Drive. Recordings are uniquely identified by (drive_path, base_name) and queryable by drive_path and/or base_name, each via exact match, prefix, or substring (see :data:MatchMode).

Attributes

MatchMode `module-attribute`

MatchMode = Literal['exact', 'prefix', 'contains']

How a filter value is matched against a catalog field: an exact match, a prefix match, or a substring match.

Classes

AmbiguousRecordingError

Bases: CatalogError

Raised when a base_name lookup matches more than one recording.

Catalog

Main interface for radiens-drive-catalog.

Wraps Google Drive scanning, local JSON catalog management, and file download. Querying is done directly on the recordings_df and items_df DataFrames using standard pandas operations.

Recordings are uniquely identified by (drive_path, base_name) where drive_path is the slash-joined path from the Drive root to the containing folder.

Example

from radiens_drive_catalog import Catalog, Config

config = Config.from_file("config.json")
catalog = Catalog(config)

catalog.scan()
hits = catalog.list_recordings(drive_path="2026-02", drive_path_mode="prefix")
path = catalog.get_recording_path("2026-02/reaching", "rat01")
catalog.prefetch(drive_path="2026-02", drive_path_mode="prefix")
print(catalog.file_tree())
print(catalog.summary())

Attributes

items_df `property`

items_df

The full Drive items catalog as a pandas DataFrame.

Columns: name, is_folder, drive_path, drive_id, mime_type, local_path, upload_time, size

recordings_df `property`

recordings_df

The full recording catalog as a pandas DataFrame.

Columns: base_name, drive_path, drive_file_ids, local_path, upload_time, size

Functions

download_item

download_item(drive_path, name)

Download a Drive item (file or folder) to the local data directory.

Items are stored under {local_data_dir}/{drive_path}/{name}.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the item's parent folder. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`name`	`str`	The file or folder name.	required

Returns:

Type	Description
`str`	The local path to the downloaded file or folder.

Raises:

Type	Description
`EntryNotFoundError`	If the item is not found in the catalog.

download_recording

download_recording(drive_path, base_name)

Download the xdat files for a recording to the local data directory.

Files are stored under {local_data_dir}/{drive_path}/.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the folder containing the recording. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`base_name`	`str`	The recording identifier (shared filename stem).	required

Returns:

Type	Description
`str`	The local directory path where the files were written.

Raises:

Type	Description
`EntryNotFoundError`	If the recording is not found in the catalog.

file_tree

file_tree()

Return a string rendering of the Drive tree with local-presence indicators.

Each entry is annotated as [recording], [folder], or [file] and [local] or [not local] based on whether it has been downloaded.

Returns:

Type	Description
`str`	A multi-line indented string representing the Drive folder hierarchy.

get_item_path

get_item_path(drive_path, name)

Return the local path for a Drive item, downloading if needed.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the item's parent folder. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`name`	`str`	The file or folder name.	required

Returns:

Type	Description
`str`	The local path to the downloaded file or folder.

Raises:

Type	Description
`EntryNotFoundError`	If the item is not found in the catalog.

get_recording

get_recording(base_name, drive_path=None)

Look up a recording by base_name, requiring it to be unique.

Parameters:

Name	Type	Description	Default
`base_name`	`str`	The recording's filename stem.	required
`drive_path`	`str \| Path \| None`	Exact `drive_path` to disambiguate recordings that share a `base_name` at different Drive locations. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	`None`

Returns:

Type	Description
`RecordingEntry`	The matching :class:`RecordingEntry`.

Raises:

Type	Description
`EntryNotFoundError`	If no recording with this base_name (and `drive_path`, if given) exists.
`AmbiguousRecordingError`	If more than one recording matches.

get_recording_path

get_recording_path(drive_path, base_name)

Return the local directory path for a recording, downloading if needed.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the folder containing the recording. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`base_name`	`str`	The recording identifier (shared filename stem).	required

Returns:

Type	Description
`str`	The local directory path where the xdat files reside.

Raises:

Type	Description
`EntryNotFoundError`	If the recording is not found in the catalog.

list_items

list_items(
    *,
    drive_path=None,
    drive_path_mode="exact",
    is_folder=None,
)

Query the Drive items catalog and return a filtered DataFrame.

All filters are applied together (AND semantics). Omitting all arguments returns the full items catalog.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path \| None`	Value to match against each item's parent `drive_path`. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	`None`
`drive_path_mode`	`MatchMode`	How `drive_path` is matched: `"exact"` (default), `"prefix"`, or `"contains"`.	`'exact'`
`is_folder`	`bool \| None`	When `True` return only folders; when `False` return only files; when `None` (default) return both.	`None`

list_recordings

list_recordings(
    *,
    base_name=None,
    base_name_mode="exact",
    drive_path=None,
    drive_path_mode="exact",
)

Query the recording catalog and return a filtered DataFrame.

All filters are applied together (AND semantics). Omitting all arguments returns the full catalog.

Parameters:

Name	Type	Description	Default
`base_name`	`str \| None`	Value to match against each recording's `base_name`.	`None`
`base_name_mode`	`MatchMode`	How `base_name` is matched: `"exact"` (default), `"prefix"`, or `"contains"`.	`'exact'`
`drive_path`	`str \| Path \| None`	Value to match against each recording's `drive_path`. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	`None`
`drive_path_mode`	`MatchMode`	How `drive_path` is matched: `"exact"` (default), `"prefix"`, or `"contains"`.	`'exact'`

local_footprint

local_footprint()

Return the total on-disk size, in bytes, of all downloaded data.

Sums each recording's own xdat files and each downloaded item's file or folder subtree. Use this to drive caller-managed eviction — e.g. release recordings until local_footprint() drops below a chosen cap.

Returns:

Type	Description
`int`	The total local footprint in bytes.

prefetch

prefetch(
    *,
    drive_path=None,
    drive_path_mode="exact",
    base_name=None,
    base_name_mode="exact",
    recordings=True,
    items=True,
    is_folder=None,
    force=False,
    max_workers=8,
)

Bulk-download matching recordings and items, skipping already-local entries.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path \| None`	Value to match against each entry's `drive_path`. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	`None`
`drive_path_mode`	`MatchMode`	How `drive_path` is matched: `"exact"` (default), `"prefix"`, or `"contains"`.	`'exact'`
`base_name`	`str \| None`	Value to match against each recording's `base_name`. Only constrains recordings — items have no `base_name` and are unaffected by this filter.	`None`
`base_name_mode`	`MatchMode`	How `base_name` is matched: `"exact"` (default), `"prefix"`, or `"contains"`.	`'exact'`
`recordings`	`bool`	When `False`, no recordings are downloaded.	`True`
`items`	`bool`	When `False`, no items are downloaded.	`True`
`is_folder`	`bool \| None`	Restrict item downloads to folders (`True`) or files (`False`).	`None`
`force`	`bool`	When `True`, re-download regardless of local presence.	`False`
`max_workers`	`int`	Number of recordings/items to download concurrently. Downloads are I/O-bound on the Drive API, which throttles each stream well below typical link bandwidth, so several concurrent streams give near-linear speedup. Each worker thread uses its own Drive service (the API client is not thread-safe). Use `1` to download strictly sequentially.	`8`

Returns:

Name	Type	Description
`A`	`PrefetchResult`	class:`PrefetchResult` with per-category download and skip counts.

Raises:

Type	Description
`Exception`	Re-raises the first download failure after recording all successful downloads in the catalog. Because prefetch is idempotent (already-local entries are skipped), re-running resumes only the failed downloads.

release_item

release_item(drive_path, name)

Delete a Drive item's local file or folder and clear its local_path.

A file item's local path is removed with unlink; a folder item's local path is an entire subtree owned by that item and is removed with shutil.rmtree. Safe to call when the item is not downloaded — it is a no-op that returns 0.

Only this item's own local_path is cleared. Deleting a folder also removes any nested files/subfolders that were cataloged as separate entries (see :meth:scan), so their stored local_path becomes stale until the next :meth:scan. This is by design: local_path is a hint reconciled by :meth:scan, and all download/size operations (:meth:get_item_path, :meth:prefetch, :meth:local_footprint) re-check the disk, so a stale value never causes a wrong result.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the item's parent folder. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`name`	`str`	The file or folder name.	required

Returns:

Type	Description
`int`	The number of bytes freed from disk.

Raises:

Type	Description
`EntryNotFoundError`	If the item is not found in the catalog.

release_recording

release_recording(drive_path, base_name)

Delete a recording's local xdat files and clear its local_path.

Only the recording's own three xdat files are removed; the containing drive_path directory is left in place because sibling recordings and downloaded items may share it. Safe to call when the recording is not downloaded — it is a no-op that returns 0.

Parameters:

Name	Type	Description	Default
`drive_path`	`str \| Path`	The slash-joined path to the folder containing the recording. Accepts a `str` or `Path`; a `Path` is converted via `str()`.	required
`base_name`	`str`	The recording identifier (shared filename stem).	required

Returns:

Type	Description
`int`	The number of bytes freed from disk.

Raises:

Type	Description
`EntryNotFoundError`	If the recording is not found in the catalog.

scan

scan(*, flat=True)

Scan Drive and rebuild the catalog JSON.

Any existing local_path entries are preserved so a rescan doesn't forget which recordings or items have already been downloaded.

Parameters:

Name	Type	Description	Default
`flat`	`bool`	If `True` (the default), use a flat scan of all files visible to the service account. Set to `False` for a recursive traversal from the root folder.	`True`

Returns:

Name	Type	Description
`A`	`ScanResult`	class:`ScanResult` with new/existing/removed counts.

summary

summary(*, rescan=False, verbose=False)

Return a human-readable report summarizing the whole catalog.

The default report is headline-only: recording/item counts, local-download progress, the upload-date range, Drive size (broken out separately for recordings and items, plus local storage size), average recording size, and an incomplete-recording count. Pass verbose=True to additionally get an item-type breakdown, the largest entries by size, the itemized incomplete-recordings list, a per-top-level-folder breakdown, and a full per-entry listing — sections whose length scales with the size of the catalog, so they're opt-in rather than pushing the headline stats out of view.

This package once had (and deliberately removed) a similar convenience method, status(), in favor of operating on the DataFrames directly. summary() is justified where status() wasn't: it's a genuine multi-source rollup across both DataFrames plus a live local-disk walk, not a one-column derivation a caller could trivially replicate.

Parameters:

Name	Type	Description	Default
`rescan`	`bool`	If `True`, call :meth:`scan` first to refresh the catalog and backfill `size` on older entries before computing the report. Defaults to `False` (fast, read-only).	`False`
`verbose`	`bool`	If `True`, additionally include the item-type breakdown, largest entries, itemized incomplete recordings, per-top-level-folder breakdown, and a full per-entry listing.	`False`

Returns:

Type	Description
`str`	A multi-line report string.

CatalogError

Bases: Exception

Base class for all radiens-drive-catalog errors.

Config `dataclass`

Configuration for radiens-drive-catalog.

All path fields (credentials_path, local_data_dir, catalog_path) support ~ and $ENV_VAR expansion and are resolved to absolute paths on construction. Typically created via Config.from_file() rather than directly.

When loaded via Config.from_file(), relative paths resolve against the config file's own directory. When constructed directly (Config(...)), relative paths resolve against the current working directory.

Attributes:

Name	Type	Description
`credentials_path`	`str`	Path to the Google service account credentials JSON file.
`root_folder_id`	`str`	Google Drive folder ID of the data root folder.
`local_data_dir`	`str`	Local directory where datasets will be downloaded.
`catalog_path`	`str`	Path to the catalog JSON file (created by `Catalog.scan()`).

Functions

__post_init__

__post_init__()

Expand ~ and $ENV_VARS in all path fields and resolve to absolute paths.

from_file `classmethod`

from_file(path=None)

Load config from a JSON file.

Resolution order when path is None:

RADIENS_DRIVE_CATALOG_CONFIG environment variable.
.secrets/config.json, then config.json, searched starting in the current working directory and then each parent directory up to the filesystem root (closest directory wins).
~/.config/radiens-drive/config.json in the user's home directory.
/etc/radiens-drive/config.json.

Parameters:

Name	Type	Description	Default
`path`	`str \| None`	Path to the config JSON file. When `None`, the resolution order above is used.	`None`

Relative paths in the config file are resolved against the config file's own directory, not the current working directory.

Returns:

Type	Description
`Config`	A `Config` instance with all paths expanded and resolved.

Raises:

Type	Description
`FileNotFoundError`	If no config file can be located.
`JSONDecodeError`	If the config file is not valid JSON.
`TypeError`	If the JSON fields do not match the `Config` field names.

DriveItemEntry

Bases: TypedDict

One non-recording Drive item stored in the catalog.

Represents a single file or folder found on Drive that is not part of an xdat recording — for example a logs/ directory, a config file, or a writeup. No structural or semantic classification is applied; the consumer decides what these items mean.

Attributes:

Name	Type	Description
`name`	`str`	The file or folder name as it appears on Drive (e.g. `"logs"`).
`is_folder`	`bool`	`True` if this item is a Drive folder.
`drive_path`	`str`	Slash-joined path from the root folder to the parent folder of this item. Together with `name`, this uniquely identifies an item.
`drive_id`	`str`	Google Drive ID of this file or folder.
`mime_type`	`str`	MIME type as reported by the Drive API.
`local_path`	`str \| None`	Absolute local path to the downloaded file or folder, or `None` if not yet downloaded.
`upload_time`	`str \| None`	ISO 8601 `createdTime` from the Drive API, or `None`.
`size`	`int \| None`	Size in bytes as reported by Drive, or `None` for folders (Drive doesn't report a size for them) and for files Drive has no size for (e.g. Google Workspace-native Docs/Sheets/Slides).

EntryNotFoundError

Bases: CatalogError

Raised when a recording or item cannot be found in the catalog.

PrefetchResult `dataclass`

Summary of a :meth:Catalog.prefetch call.

Attributes:

Name	Type	Description
`recordings_downloaded`	`int`	Number of recordings fetched from Drive.
`recordings_skipped`	`int`	Number of recordings already available locally.
`items_downloaded`	`int`	Number of items fetched from Drive.
`items_skipped`	`int`	Number of items already available locally.

RecordingEntry

Bases: TypedDict

One xdat recording dataset stored in the catalog.

Represents a single neural recording. The three xdat files (_data.xdat, .xdat.json, _timestamp.xdat) share a common base_name stem and are treated as a single unit.

Attributes:

Name	Type	Description
`base_name`	`str`	Shared filename stem across all three xdat files. Note: `base_name` is not globally unique; the pair `(drive_path, base_name)` uniquely identifies a recording.
`drive_path`	`str`	Slash-joined path from the root folder to the folder containing the recording files.
`drive_file_ids`	`dict[str, str]`	Maps file type labels (`"data"`, `"meta"`, `"timestamp"`) to their Google Drive file IDs.
`local_path`	`str \| None`	Absolute path to the local directory where the recording has been downloaded, or `None` if not yet downloaded.
`upload_time`	`str \| None`	ISO 8601 `createdTime` from the Drive API, or `None`.
`size`	`int \| None`	Combined size in bytes of the constituent xdat files that reported a size (summed from whichever of the three files Drive reported a size for). `None` only if none of them did.

ScanResult `dataclass`

Summary of a :meth:Catalog.scan call.

Attributes:

Name	Type	Description
`recordings_new`	`int`	Recordings found on Drive not in the previous catalog.
`recordings_existing`	`int`	Recordings found on Drive already in the catalog.
`recordings_removed`	`int`	Recordings in the previous catalog not on Drive.
`items_new`	`int`	Items found on Drive not in the previous catalog.
`items_existing`	`int`	Items found on Drive already in the catalog.
`items_removed`	`int`	Items in the previous catalog not found on Drive.

Functions

match_series

match_series(series, query, mode)

Filter a string pd.Series by query under mode, returning a boolean mask.

Use this to filter an already-fetched DataFrame (e.g. a cached recordings_df, or your own subset of it) with the same exact/prefix/contains semantics :meth:Catalog.list_recordings and :meth:Catalog.list_items use internally. A None query matches everything (an all-True mask).

radiens_drive_catalog