radiens_drive_catalog

radiens-drive-catalog

Programmatic catalog and sync tool for xdat neural datasets stored on Google Drive. Datasets are indexed by base_name and queryable by date folder, experiment, and sub-experiment.

Classes

AssetEntry

Bases: TypedDict

One non-xdat asset record stored in the catalog.

Represents a single file or folder found on Drive that is not an xdat dataset — for example a logs/ directory, a PowerPoint, or a writeup.

Attributes:

Name	Type	Description
asset_name	str	The file or folder name as it appears on Drive (e.g. "logs", "notes.pptx").
asset_type	Literal['folder', 'file']	"folder" or "file".
date_folder	str | None	Name of the top-level date-prefixed folder (depth 1), or None if the asset is at the Drive root.
experiment	str | None	Name of the depth-2 subfolder, or None if the asset lives directly inside the date folder.
drive_path	str	Slash-joined path from the root folder to the parent folder of this asset (e.g. "2026-02-15_batch/reaching").
drive_id	str	Google Drive ID of this file or folder.
mime_type	str	MIME type as reported by the Drive API.
local_path	str | None	Absolute local path to the downloaded file or folder, or None if not yet downloaded.

Catalog

Main interface for radiens-drive-catalog.

Wraps Google Drive scanning, local JSON catalog management, dataset and asset querying, and file download behind a single object.

Example

from radiens_drive_catalog import Catalog, Config

config = Config.from_file("config.json")
catalog = Catalog(config)

catalog.scan()                              # build catalog from Drive
df = catalog.list(date="2026-02")           # query datasets
adf = catalog.list_assets(experiment="reaching")  # query assets
path = catalog.get_path("rat01_session3")  # download dataset if needed
path = catalog.get_asset_path("2026-02-15/reaching", "logs")  # download asset if needed

Attributes

assets_df property

assets_df

The full assets catalog as a pandas DataFrame. Columns: asset_name, asset_type, date_folder, experiment, drive_path, drive_id, mime_type, local_path

df property

df

The full dataset catalog as a pandas DataFrame. Columns: base_name, date_folder, experiment, drive_path, drive_file_ids, local_path

Functions

__init__

__init__(config)

Initialize a Catalog.

Parameters:

Name	Type	Description	Default
config	Config	Package configuration (Drive credentials, folder ID, local paths).	required

download

download(base_name)

Download the xdat files for a dataset to local_data_dir.

After a successful download, persists the local_path back to the catalog JSON so it survives future sessions.

Parameters:

Name	Type	Description	Default
base_name	str	The dataset identifier (shared filename stem).	required

Returns:

Type	Description
str	The local directory path where the files were written.

Raises:

Type	Description
ValueError	If base_name is not found in the catalog.

download_asset

download_asset(drive_path, asset_name)

Download an asset (file or folder) to the local assets directory.

Assets are stored under {local_data_dir}/assets/{drive_path}/{asset_name}. For folder assets the entire subtree is downloaded recursively.

After a successful download, persists the local_path back to the catalog JSON.

Parameters:

Name	Type	Description	Default
drive_path	str	The slash-joined path to the asset's parent folder (e.g. "2026-02-15_batch/reaching"). Shown in assets_df["drive_path"].	required
asset_name	str	The file or folder name (e.g. "logs").	required

Returns:

Type	Description
str	The local path to the downloaded file or folder.

Raises:

Type	Description
ValueError	If the asset is not found in the catalog.

get_asset_path

get_asset_path(drive_path, asset_name)

Return the local path for an asset, downloading if needed.

If the asset has not been downloaded yet, or if the recorded local_path no longer exists on disk, the download is triggered automatically.

Parameters:

Name	Type	Description	Default
drive_path	str	The slash-joined path to the asset's parent folder (e.g. "2026-02-15_batch/reaching").	required
asset_name	str	The file or folder name (e.g. "logs").	required

Returns:

Type	Description
str	The local path to the downloaded file or folder.

Raises:

Type	Description
ValueError	If the asset is not found in the catalog.

get_path

get_path(base_name)

Return the local directory path for a dataset, downloading if needed.

If the dataset has not been downloaded yet, or if the recorded local_path no longer exists on disk, the download is triggered automatically.

Parameters:

Name	Type	Description	Default
base_name	str	The dataset identifier (shared filename stem).	required

Returns:

Type	Description
str	The local directory path where the xdat files reside.

Raises:

Type	Description
ValueError	If base_name is not found in the catalog.

list

list(date_folder=None, experiment=None)

Query the dataset catalog and return a filtered DataFrame.

Arguments act as hierarchical filters — each narrows the result further. Omitting an argument returns all values for that level.

Parameters:

Name	Type	Description	Default
date_folder	str | None	Exact name of the top-level date folder to filter on (e.g. "2026-02-15_batch"). Must match the full folder name.	None
experiment	str | None	Exact name of the depth-2 experiment folder to filter on.	None

Examples:

catalog.list() # everything catalog.list(date_folder="2026-02-15_batch") # one date folder catalog.list(date_folder="2026-02-15_batch", experiment="reaching") catalog.list(experiment="reaching") # across all dates

list_assets

list_assets(
    date_folder=None, experiment=None, asset_type=None
)

Query the asset catalog and return a filtered DataFrame.

Arguments act as hierarchical filters. Omitting an argument returns all values for that level.

Parameters:

Name	Type	Description	Default
date_folder	str | None	Exact name of the top-level date folder to filter on (e.g. "2026-02-15_batch"). Must match the full folder name.	None
experiment	str | None	Exact name of the depth-2 experiment folder to filter on.	None
asset_type	Literal['folder', 'file'] | None	"folder" or "file"; if omitted, returns both.	None

Examples:

catalog.list_assets() catalog.list_assets(date_folder="2026-02-15_batch", experiment="reaching") catalog.list_assets(asset_type="folder")

scan

scan()

Scan Drive recursively and rebuild the catalog JSON.

Any existing local_path entries are preserved so a rescan doesn't forget which datasets or assets have already been downloaded.

status

status()

Return the dataset catalog DataFrame with an extra boolean column 'is_local' indicating which datasets are already downloaded. Useful for a quick overview of what's available locally.

Config dataclass

Configuration for radiens-drive-catalog.

All path fields (credentials_path, local_data_dir, catalog_path) support ~ and $ENV_VAR expansion and are resolved to absolute paths on construction. Typically created via Config.from_file() rather than directly.

Attributes:

Name	Type	Description
credentials_path	str	Path to the Google service account credentials JSON file.
root_folder_id	str	Google Drive folder ID of the data root folder.
local_data_dir	str	Local directory where datasets will be downloaded.
catalog_path	str	Path to the catalog JSON file (created by Catalog.scan()).

Functions

__post_init__

__post_init__()

Expand ~ and $ENV_VARS in all path fields and resolve to absolute paths.

from_file classmethod

from_file(path=None)

Load config from a JSON file.

Resolution order when path is None:

1. RADIENS_DRIVE_CATALOG_CONFIG environment variable.
2. .secrets/config.json in the current working directory.
3. config.json in the current working directory.

Parameters:

Name	Type	Description	Default
path	str | None	Path to the config JSON file. When None, the resolution order above is used.	None

Returns:

Type	Description
Config	A Config instance with all paths expanded and resolved.

Raises:

Type	Description
FileNotFoundError	If no config file can be located.
JSONDecodeError	If the config file is not valid JSON.
TypeError	If the JSON fields do not match the Config field names.