Skip to content

Configuration

Config file format

The package is configured via a JSON file with four fields:

{
    "credentials_path": "/path/to/service_account.json",
    "root_folder_id": "1aBcDeFgHiJkLmNoPqRsTuVw",
    "local_data_dir": "/data/neural",
    "catalog_path": "/data/neural/catalog.json"
}
Field Type Description
credentials_path str Path to the Google service account credentials JSON file
root_folder_id str Google Drive folder ID of the root data folder
local_data_dir str Local directory where datasets will be downloaded
catalog_path str Path to the catalog JSON file (created by scan())

The root_folder_id is the alphanumeric string in the Drive URL when you navigate to the root data folder: https://drive.google.com/drive/folders/1aBcDeFgHiJkLmNoPqRsTuVw

Loading config

Config.from_file() locates the config file using this resolution order (first match wins):

  1. Explicit path argument.
  2. RADIENS_DRIVE_CATALOG_CONFIG environment variable.
  3. .secrets/config.json, then config.json, searched starting in the current working directory and then each parent directory up to the filesystem root — the closest directory wins. This mirrors how git locates .git: discovery works the same whether you run from the repo root or a nested subdirectory (e.g. a notebook in notebooks/).
  4. ~/.config/radiens-drive/config.json.
  5. /etc/radiens-drive/config.json.
# Automatic discovery — uses env var or well-known paths
config = Config.from_file()

# Explicit path
config = Config.from_file("/path/to/config.json")

For most projects, placing the config at config.json in the repository root (with credentials and other sensitive files kept in a .secrets/ directory that's .gitignored, referenced by path from config.json) is sufficient — no environment variable needed. See the note on relative paths below before choosing between config.json and .secrets/config.json for the config file's own location.

Path expansion

All path fields (credentials_path, local_data_dir, catalog_path) support:

  • ~ — expanded to the current user's home directory
  • $ENV_VAR — expanded from the environment
{
    "credentials_path": "~/secrets/service_account.json",
    "local_data_dir": "$LAB_DATA/neural",
    "catalog_path": "$LAB_DATA/neural/catalog.json",
    "root_folder_id": "1aBcDeFgHiJkLmNoPqRsTuVw"
}

Paths are resolved to absolute paths when the Config object is created. When loaded via Config.from_file(), relative paths are resolved against the directory containing the config file itself — not the process's current working directory. This is the same convention used by tools like tsconfig.json: a relative path in the config is always anchored to where the config file lives, so the same config produces the same absolute paths no matter where the calling script or notebook happens to run from.

This matters for where you place config.json:

{
    "credentials_path": "service_account.json",
    "local_data_dir": "../data/drive-data",
    "catalog_path": "../data/drive-data/catalog.json",
    "root_folder_id": "1aBcDeFgHiJkLmNoPqRsTuVw"
}

If this file is saved as <repo>/config.json, local_data_dir resolves to <repo>/../data/drive-data — probably not what you want. Relative paths are always relative to this file's own location, so:

  • If config.json sits directly in the repo root, write paths relative to the repo root directly (no .secrets/ prefix needed on credentials_path if credentials live there too — "credentials_path": ".secrets/service_account.json", "local_data_dir": "data/drive-data").
  • If config.json itself sits inside .secrets/ (e.g. discovered via the .secrets/config.json well-known path), paths need an extra ../ to reach the repo root, and sibling files in .secrets/ need no prefix at all ("credentials_path": "service_account.json").

When in doubt, keep config.json at the repo root — it keeps every relative path in it a direct match for the repo layout.

(Constructing a Config directly, rather than via from_file(), has no file to anchor to, so relative paths resolve against the current working directory instead.)

Service account setup

This package uses a Google service account so all collaborators share the same credentials without needing individual Google accounts.

  1. Create a project in Google Cloud Console
  2. Enable the Google Drive API for the project
  3. Create a Service Account under IAM & Admin → Service Accounts
  4. Generate a JSON key for the service account and download it
  5. Share your root Drive data folder with the service account's email address (Viewer access is sufficient)
  6. Set credentials_path in your config to point at the downloaded JSON file

Security

Warning

The credentials JSON file grants read access to any Drive folder shared with the service account. Treat it like a password.

  • Do not commit the credentials file to version control
  • Add it to .gitignore:
service_account.json
*_credentials.json
  • Distribute it to collaborators via a secure channel (e.g. an encrypted password manager or a private shared folder)