Imaging Data Commons

When to Use

• Use this skill when you need use idc-index to query and download public cancer imaging data from nci imaging data commons. used to access large-scale radiology (ct, mr, pet) and pathology datasets for ai training or research. no authentication required. supports metadata querying, in-browser visualization, and license checking in a reproducible workflow.
• Use this skill when a data analytics task needs a packaged method instead of ad-hoc freeform output.
• Use this skill when the user expects a concrete deliverable, validation step, or file-based result.
• Use this skill when the documented workflow in this package is the most direct path to complete the request.
• Use this skill when you need the imaging-data-commons package behavior rather than a generic answer.

Key Features

• Scope-focused workflow aligned to: Use idc-index to query and download public cancer imaging data from NCI Imaging Data Commons. Used to access large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Supports metadata querying, in-browser visualization, and license checking.
• Documentation-first workflow with no packaged script requirement.
• Reference material available in references/ for task-specific guidance.
• Structured execution path designed to keep outputs consistent and reviewable.

Dependencies

• Python: 3.10+. Repository baseline for current packaged skills.
• Third-party packages: not explicitly version-pinned in this skill package. Add pinned versions if this skill needs stricter environment control.

Example Usage

Skill directory: 20260316/scientific-skills/Data Analytics/imaging-data-commons
No packaged executable script was detected.
Use the documented workflow in SKILL.md together with the references/assets in this folder.

Example run plan:

1. Read the skill instructions and collect the required inputs.
2. Follow the documented workflow exactly.
3. Use packaged references/assets from this folder when the task needs templates or rules.
4. Return a structured result tied to the requested deliverable.

Implementation Details

See ## Overview above for related details.

• Execution model: validate the request, choose the packaged workflow, and produce a bounded deliverable.
• Input controls: confirm the source files, scope limits, output format, and acceptance criteria before running any script.
• Primary implementation surface: instruction-only workflow in SKILL.md.
• Reference guidance: references/ contains supporting rules, prompts, or checklists.
• Parameters to clarify first: input path, output path, scope filters, thresholds, and any domain-specific constraints.
• Output discipline: keep results reproducible, identify assumptions explicitly, and avoid undocumented side effects.

Overview

Use the idc-index Python package to query and download public cancer imaging data from the National Cancer Institute (NCI) Imaging Data Commons (IDC). No authentication required to access data.

Core Tool: idc-index (GitHub)

View latest data scale:

from idc_index import IDCClient
client = IDCClient()

# Get IDC data version
print(client.get_idc_version())

# Get collection count and total series count
stats = client.sql_query("""
    SELECT   
        COUNT(DISTINCT collection_id) as collections,
        COUNT(DISTINCT analysis_result_id) as analysis_results,
        COUNT(DISTINCT PatientID) as patients,
        COUNT(DISTINCT StudyInstanceUID) as studies,
        COUNT(DISTINCT SeriesInstanceUID) as series,
        SUM(instanceCount) as instances,
        SUM(series_size_MB)/1000000 as size_TB
    FROM index
""")
print(stats)

Core Workflow:

1. Query metadata → client.sql_query()
2. Download DICOM files → client.download_from_selection()
3. Visualize in browser → client.get_viewer_URL(seriesInstanceUID=...)

When to Use This Skill

• Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
• Filtering image subsets by cancer type, imaging modality, anatomical site, or other metadata
• Downloading DICOM data from IDC
• Checking data licenses before research or commercial use
• Visualizing medical images in the browser without needing local DICOM viewer software

IDC Data Model

IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):

• collection_id: Groups patients by disease, modality, or research focus (e.g., tcga_luad, nlst). A patient belongs to only one collection.
• analysis_result_id: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.

Use collection_id to find original imaging data (may include annotations stored with images); use analysis_result_id to find AI-generated or expert-annotated annotations.

Key Identifiers for Querying:

Identifier	Scope	Purpose
collection_id	Dataset grouping	Filter by project/study
PatientID	Patient	Group images by patient
StudyInstanceUID	DICOM study	Group related series, visualization
SeriesInstanceUID	DICOM series	Group related series, visualization

Index Tables

The idc-index package provides multiple metadata index tables accessible via SQL or pandas DataFrame.

Important: Use client.indices_overview to get current table descriptions and column schemas. This is the authoritative source for available columns and their types—always consult this when writing SQL or exploring data structures.

Available Tables

Table Name	Row Granularity	Loading Method	Description
index	1 row = 1 DICOM series	Auto	Core metadata for all current IDC data
prior_versions_index	1 row = 1 DICOM series	Auto	Series from previous IDC versions; for downloading deprecated data
collections_index	1 row = 1 collection	fetch_index()	Collection-level metadata and descriptions
analysis_results_index	1 row = 1 analysis result set	fetch_index()	Metadata about derived datasets (annotations, segmentations)
clinical_index	1 row = 1 clinical data column	fetch_index()	Mapping of clinical table columns to collections
sm_index	1 row = 1 slide microscopy series	fetch_index()	Slide microscopy (pathology) series metadata
sm_instance_index	1 row = 1 slide microscopy instance	fetch_index()	Instance-level metadata for slide microscopy (SOPInstanceUID)
seg_index	1 row = 1 DICOM segmentation series	fetch_index()	Segmentation metadata: algorithm, number of segments, reference to source image series

Auto = Automatically loaded when IDCClient() is instantiated fetch_index() = Requires executing client.fetch_index("table_name") to load

Joining Tables

Key columns are not explicitly marked; here is a subset that can be used for joins.

Join Column	Tables	Use Case
collection_id	index, prior_versions_index, collections_index, clinical_index	Link series to collection metadata or clinical data
SeriesInstanceUID	index, prior_versions_index, sm_index, sm_instance_index	Cross-table linking of series; connect to slide microscopy details
StudyInstanceUID	index, prior_versions_index	Link studies across current and historical data
PatientID	index, prior_versions_index	Link patients across current and historical data
analysis_result_id	index, analysis_results_index	Link series to analysis result metadata (annotations, segmentations)
source_DOI	index, analysis_results_index	Link via publication DOI
crdc_series_uuid	index, prior_versions_index	Link via CRDC unique identifier
Modality	index, prior_versions_index	Filter by imaging modality
SeriesInstanceUID	index, seg_index	Link segmentation series to its index metadata
segmented_SeriesInstanceUID	seg_index → index	Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID)

Note: Subjects, Updated, and Description appear in multiple tables but with different meanings (counts vs. identifiers, different update contexts).

Join Examples:

from idc_index import IDCClient
client = IDCClient()

# Join index with collections_index to get cancer types
client.fetch_index("collections_index")
result = client.sql_query("""
    SELECT i.SeriesInstanceUID, i.Modality, c.CancerTypes, c.TumorLocations
    FROM index i
    JOIN collections_index c ON i.collection_id = c.collection_id
    WHERE i.Modality = 'MR'
    LIMIT 10
""")

# Join index with sm_index to get slide microscopy details
client.fetch_index("sm_index")
result = client.sql_query("""
    SELECT i.collection_id, i.PatientID, s.ObjectiveLensPower, s.min_PixelSpacing_2sf
    FROM index i
    JOIN sm_index s ON i.SeriesInstanceUID = s.SeriesInstanceUID
    LIMIT 10
""")

# Join seg_index with index to find segmentations and their source images
client.fetch_index("seg_index")
result = client.sql_query("""
    SELECT
        s.SeriesInstanceUID as seg_series,
        s.AlgorithmName,
        s.total_segments,
        src.collection_id,
        src.Modality as source_modality,
        src.BodyPartExamined
    FROM seg_index s
    JOIN index src ON s.segmented_SeriesInstanceUID = src.SeriesInstanceUID
    WHERE s.AlgorithmType = 'AUTOMATIC'
    LIMIT 10
""")

Accessing Index Tables

Via SQL (recommended for filtering/aggregation):

from idc_index import IDCClient
client = IDCClient()

# Query main index (always available)
results = client.sql_query("SELECT * FROM index WHERE Modality = 'CT' LIMIT 10")

# Fetch and query additional indexes
client.fetch_index("collections_index")
collections = client.sql_query("SELECT collection_id, CancerTypes, TumorLocations FROM collections_index")

client.fetch_index("analysis_results_index")
analysis = client.sql_query("SELECT * FROM analysis_results_index LIMIT 5")

As pandas DataFrame (direct access):

# Main index (always available after client initialization)
df = client.index

# Fetch and access on-demand indexes
client.fetch_index("sm_index")
sm_df = client.sm_index

Discovering Table Schemas (Key to Writing Queries)

The indices_overview dictionary contains complete schema information for all tables. Always refer to this information when writing queries or exploring data structures.

DICOM Attribute Mapping: Many columns are populated directly from DICOM attributes in source files. Column descriptions in the schema indicate whether a column corresponds to a DICOM attribute (e.g., "DICOM Modality attribute" or reference to a DICOM tag). This allows leveraging DICOM knowledge at query time—standard DICOM attribute names like PatientID, StudyInstanceUID, Modality, BodyPartExamined all work as expected.

from idc_index import IDCClient
client = IDCClient()

# List all available indexes with their descriptions
for name, info in client.indices_overview.items():
    print(f"\n{name}:")
    print(f"  Installed: {info['installed']}")
    print(f"  Description: {info['description']}")

# Get full schema for a specific index (columns, types, descriptions)
schema = client.indices_overview["index"]["schema"]
print(f"\nTable: {schema['table_description']}")
print("\nColumns:")
for col in schema['columns']:
    desc = col.get('description', 'No description')
    # Description indicates whether column comes from DICOM attribute
    print(f"  {col['name']} ({col['type']}): {desc}")

# Find columns that belong to DICOM attributes (check if description contains "DICOM")
dicom_cols = [c['name'] for c in schema['columns'] if 'DICOM' in c.get('description', '').upper()]
print(f"\nDICOM-sourced columns: {dicom_cols}")

Alternative: Use get_index_schema() method:

schema = client.get_index_schema("index")

# Returns the same schema dictionary: {'table_description': ..., 'columns': [...]}

Key Columns in Primary index Table

Most commonly used columns in queries (full list and descriptions available in indices_overview):

Column Name	Type	DICOM	Description
collection_id	STRING	No	IDC collection identifier
analysis_result_id	STRING	No	If applicable, indicates the analysis result set this series belongs to
source_DOI	STRING	No	DOI linking to dataset details; for further content info and attribution (see citation section below)
PatientID	STRING	Yes	Patient identifier
StudyInstanceUID	STRING	Yes	DICOM study UID
SeriesInstanceUID	STRING	Yes	DICOM series UID — for download/viewing
Modality	STRING	Yes	Imaging modality (CT, MR, PT, SM, etc.)
BodyPartExamined	STRING	Yes	Anatomical site
SeriesDescription	STRING	Yes	Series description
Manufacturer	STRING	Yes	Device manufacturer
StudyDate	STRING	Yes	Study execution date
PatientSex	STRING	Yes	Patient sex
PatientAge	STRING	Yes	Patient age at time of study
license_short_name	STRING	No	License type (CC BY 4.0, CC BY-NC 4.0, etc.)
series_size_MB	FLOAT	No	Series size in MB
instanceCount	INTEGER	No	Number of DICOM instances in the series

DICOM = Yes: Column values extracted from DICOM attributes of the same name. For numeric tag mappings, refer to the DICOM Standard. Use standard DICOM knowledge to anticipate values and formats.

Clinical Data Access

# Fetch clinical index (also downloads clinical data tables)
client.fetch_index("clinical_index")

# Query clinical index to find available tables and their columns
tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")

# Load specific clinical table as DataFrame
clinical_df = client.get_clinical_table("table_name")

For detailed workflows including value mapping schemas and joining clinical data with imaging data, see references/clinical_data_guide.md.

Data Access Options

Method	Authentication Required	Best For
idc-index	No	Key queries and downloads (recommended)
IDC Portal	No	Interactive exploration, manual selection, browser-based download
BigQuery	Yes (GCP account)	Complex queries, complete DICOM metadata
DICOMweb proxy	No	Tool integration via DICOMweb API
Cloud storage (S3/GCS)	No	Direct file access, batch downloads, custom pipelines

Cloud Storage Organization

All DICOM files mirrored between AWS S3 and Google Cloud Storage are stored in public cloud storage buckets. Files are organized by CRDC UUID (not DICOM UID) to support versioning.

Bucket (AWS / GCS)	License	Content
idc-open-data / idc-open-data	No commercial restrictions	>90% of IDC data
idc-open-data-two / idc-open-idc1	No commercial restrictions	Collections that may contain head scans
idc-open-data-cr / idc-open-cr	Commercial use restricted (CC BY-NC)	~4% of data

File storage format is <crdc_series_uuid>/<crdc_instance_uuid>.dcm. Freely accessible via AWS CLI, gsutil, or s5cmd with anonymous access (no bandwidth fees). Use the series_aws_url column in the index to get S3 URL; GCS uses the same path structure.

For bucket details, access commands, UUID mapping, and versioning, see references/cloud_storage_guide.md.

DICOMweb Access

IDC data is available via DICOMweb interface (implemented on Google Cloud Healthcare API) for integrating PACS systems and DICOMweb-compatible tools.

Endpoint	Authentication	Use Case
Public proxy	No	Testing, moderate queries, daily quota
Google Healthcare	Yes (GCP)	Production use, higher quotas

For endpoint URLs, code examples, supported operations, and implementation details, see references/dicomweb_guide.md.

Installation and Setup

Required (basic access):

pip install --upgrade idc-index

Important: Each IDC data release triggers a new version of idc-index. Unless you need an older version for reproducibility, always use the --upgrade flag when installing.

Tested version: idc-index 0.11.7 (IDC data version v23)

Optional (data analysis):

pip install pandas numpy pydicom

Core Capabilities

1. Data Discovery and Exploration

Discover available imaging collections and data in IDC:

from idc_index import IDCClient

client = IDCClient()

# Get summary statistics from main index
query = """
SELECT
  collection_id,
  COUNT(DISTINCT PatientID) as patients,
  COUNT(DISTINCT SeriesInstanceUID) as series,
  SUM(series_size_MB) as size_mb
FROM index
GROUP BY collection_id
ORDER BY patients DESC
"""
collections_summary = client.sql_query(query)

# For richer collection metadata, use collections_index
client.fetch_index("collections_index")
collections_info = client.sql_query("""
    SELECT collection_id, CancerTypes, TumorLocations, Species, Subjects, SupportingData
    FROM collections_index
""")

# For analysis results (annotations, segmentations), use analysis_results_index
client.fetch_index("analysis_results_index")
analysis_info = client.sql_query("""
    SELECT analysis_result_id, analysis_result_title, Subjects, Collections, Modalities
    FROM analysis_results_index
""")

collections_index provides curated metadata for each collection: cancer types, tumor locations, species, subject counts, and supported data types, without needing aggregation from the main index.

analysis_results_index lists derived datasets (AI segmentations, expert annotations, radiomics features) with their source collections and modalities.

2. Querying Metadata with SQL

Use SQL to query the IDC mini-index to find specific datasets.

First, explore optional filter values:

from idc_index import IDCClient

client = IDCClient()

# Check what Modality values exist
modalities = client.sql_query("""
    SELECT DISTINCT Modality, COUNT(*) as series_count
    FROM index
    GROUP BY Modality
    ORDER BY series_count DESC
""")
print(modalities)

# Check what BodyPartExamined values exist for MR modality
body_parts = client.sql_query("""
    SELECT DISTINCT BodyPartExamined, COUNT(*) as series_count
    FROM index
    WHERE Modality = 'MR' AND BodyPartExamined IS NOT NULL
    GROUP BY BodyPartExamined
    ORDER BY series_count DESC
    LIMIT 20
""")
print(body_parts)

Then query using validated filter values:

# Find breast MRI scans (using actual values from exploration above)
results = client.sql_query("""
    SELECT
      collection_id,
      PatientID,
      SeriesInstanceUID,
      Modality,
      SeriesDescription,
      license_short_name
    FROM index
    WHERE Modality = 'MR'
      AND BodyPartExamined = 'BREAST'
    LIMIT 20
""")

# Access results as pandas DataFrame
for idx, row in results.iterrows():
    print(f"Patient: {row['PatientID']}, Series: {row['SeriesInstanceUID']}")

Filter by cancer type, requires joining with collections_index:

client.fetch_index("collections_index")
results = client.sql_query("""
    SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
    FROM index i
    JOIN collections_index c ON i.collection_id = c.collection_id
    WHERE c.CancerTypes LIKE '%Breast%'
      AND i.Modality = 'MR'
    LIMIT 20
""")

Available metadata fields (full list use client.indices_overview):

• Identifiers: collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID
• Imaging: Modality, BodyPartExamined, Manufacturer, ManufacturerModelName
• Clinical: PatientAge, PatientSex, StudyDate
• Descriptions: StudyDescription, SeriesDescription
• License: license_short_name

Note: Cancer types are in collections_index.CancerTypes, not in the main index table.

3. Downloading DICOM Files

Efficiently download imaging data from IDC cloud storage:

Download entire collection:

from idc_index import IDCClient

client = IDCClient()

# Download small collection (RIDER Pilot ~1GB)
client.download_from_selection(
    collection_id="rider_pilot",
    downloadDir="./data/rider"
)

Download specific series:

# First, query for series UIDs
series_df = client.sql_query("""
    SELECT SeriesInstanceUID
    FROM index
    WHERE Modality = 'CT'
      AND BodyPartExamined = 'CHEST'
      AND collection_id = 'nlst'
    LIMIT 5
""")

# Download only these series
client.download_from_selection(
    seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
    downloadDir="./data/lung_ct"
)

Custom directory structure:

Default dirTemplate is: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID

# Simplified hierarchy (omit StudyInstanceUID level)
client.download_from_selection(
    collection_id="tcga_luad",
    downloadDir="./data",
    dirTemplate="%collection_id/%PatientID/%Modality"
)

# Result path: ./data/tcga_luad/TCGA-05-4244/CT/

# Flat structure (all files in same directory)
client.download_from_selection(
    seriesInstanceUID=list(series_df['SeriesInstanceUID'].values),
    downloadDir="./data/flat",
    dirTemplate=""
)

# Result path: ./data/flat/*.dcm

Command-Line Download

The idc download command provides command-line access to download functionality without writing Python code. Available after installing idc-index.

Auto-detect input type: Manifest file path, or identifier (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).

# Download entire collection
idc download rider_pilot --download-dir ./data

# Download specific series by UID
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data

# Download multiple items (comma-separated)
idc download "tcga_luad,tcga_lusc" --download-dir ./data

# Download from manifest file (auto-detect)
idc download manifest.txt --download-dir ./data

Options:

Option	Description
--download-dir	Output directory (default: current directory)
--dir-template	Directory hierarchy template (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID)
--log-level	Logging verbosity: debug, info, warning, error, critical

Manifest files:

Manifest files contain S3 URLs (one per line), which can be:

• Exported from IDC Portal after queue selection
• Shared by collaborators for reproducible data access
• Programmatically generated from query results

Format (one S3 URL per line):

s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*

Example: Generate manifest from Python query:

from idc_index import IDCClient

client = IDCClient()

# Query series URLs
results = client.sql_query("""
    SELECT series_aws_url
    FROM index
    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
""")

# Save as manifest file
with open('ct_manifest.txt', 'w') as f:
    for url in results['series_aws_url']:
        f.write(url + '\n')

Then download:

idc download ct_manifest.txt --download-dir ./ct_data

4. Visualizing IDC Images

View DICOM data in the browser without downloading:

from idc_index import IDCClient
import webbrowser

client = IDCClient()

# First query to get valid UIDs
results = client.sql_query("""
    SELECT SeriesInstanceUID, StudyInstanceUID
    FROM index
    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
    LIMIT 1
""")

# View single series
viewer_url = client.get_viewer_URL(seriesInstanceUID=results.iloc[0]['SeriesInstanceUID'])
webbrowser.open(viewer_url)

# View all series in a study (useful for multi-series exams like MRI protocols)
viewer_url = client.get_viewer_URL(studyInstanceUID=results.iloc[0]['StudyInstanceUID'])
webbrowser.open(viewer_url)

This method automatically selects OHIF v3 for radiology or SLIM for slide microscopy. Viewing by study is very useful when a DICOM study contains multiple series (e.g., T1, T2, DWI sequences in a single MRI session).

5. Understanding and Checking Licenses

Check data licenses before use (critical for commercial applications):

from idc_index import IDCClient

client = IDCClient()

# Check licenses for all collections
query = """
SELECT DISTINCT
  collection_id,
  license_short_name,
  COUNT(DISTINCT SeriesInstanceUID) as series_count
FROM index
GROUP BY collection_id, license_short_name
ORDER BY collection_id
"""

licenses = client.sql_query(query)
print(licenses)

License types in IDC:

• CC BY 4.0 / CC BY 3.0 (~97% of data) - Allows commercial use with attribution
• CC BY-NC 4.0 / CC BY-NC 3.0 (~3% of data) - Non-commercial use only
• Custom license (rare) - Some collections have specific terms (e.g., NLM terms)

Important: Always check the license before using IDC data in publications or commercial applications. Each DICOM file's metadata tags its specific license.

Generating Citations for Attribution

The source_DOI column contains DOIs pointing to publications describing the data generation process. To meet attribution requirements, use citations_from_selection() to generate properly formatted citations:

from idc_index import IDCClient

client = IDCClient()

# Get citations for collection (APA format by default)
citations = client.citations_from_selection(collection_id="rider_pilot")
for citation in citations:
    print(citation)

# Get citations for specific series
results = client.sql_query("""
    SELECT SeriesInstanceUID FROM index
    WHERE collection_id = 'tcga_luad' LIMIT 5
""")
citations = client.citations_from_selection(
    seriesInstanceUID=list(results['SeriesInstanceUID'].values)
)

# Alternative format: BibTeX (for LaTeX documents)
bibtex_citations = client.citations_from_selection(
    collection_id="tcga_luad",
    citation_format=IDCClient.CITATION_FORMAT_BIBTEX
)

Parameters:

• collection_id: Filter by collection
• patientId: Filter by patient ID
• studyInstanceUID: Filter by study UID
• seriesInstanceUID: Filter by series UID
• citation_format: Use IDCClient.CITATION_FORMAT_* constants:
  • CITATION_FORMAT_APA (default) - APA style
  • CITATION_FORMAT_BIBTEX - BibTeX for LaTeX
  • CITATION_FORMAT_JSON - CSL JSON
  • CITATION_FORMAT_TURTLE - RDF Turtle

Best Practice: When publishing results obtained using IDC data, include generated citations to properly attribute data sources and meet license requirements.

6. Batch Processing and Filtering

Efficiently process large-scale datasets through filtering:

from idc_index import IDCClient
import pandas as pd

client = IDCClient()

# Find chest CT scans from GE scanners
query = """
SELECT
  SeriesInstanceUID,
  PatientID,
  collection_id,
  ManufacturerModelName
FROM index
WHERE Modality = 'CT'
  AND BodyPartExamined = 'CHEST'
  AND Manufacturer = 'GE MEDICAL SYSTEMS'
  AND license_short_name = 'CC BY 4.0'
LIMIT 100
"""

results = client.sql_query(query)

# Save manifest for later use
results.to_csv('lung_ct_manifest.csv', index=False)

# Download in batches to avoid timeouts
batch_size = 10
for i in range(0, len(results), batch_size):
    batch = results.iloc[i:i+batch_size]
    client.download_from_selection(
        seriesInstanceUID=list(batch['SeriesInstanceUID'].values),
        downloadDir=f"./data/batch_{i//batch_size}"
    )

7. Advanced Queries with BigQuery

For queries requiring complete DICOM metadata, complex JOINs, clinical data tables, or private DICOM elements, use Google BigQuery. Requires a GCP project with billing enabled.

Quick Reference:

• Dataset: bigquery-public-data.idc_current.*
• Main table: dicom_all (merged metadata)
• Complete metadata: dicom_metadata (all DICOM tags)
• Private elements: OtherElements column (vendor-specific tags like diffusion b-values)

For setup, table schemas, query patterns, private element access, and cost optimization, see references/bigquery_guide.md.

8. Tool Selection Guide

Task	Tool	Reference
Programmatic query and download	idc-index	This document
Interactive exploration	IDC Portal	https://portal.imaging.datacommons.cancer.gov/
Complex metadata queries	BigQuery	references/bigquery_guide.md
3D visualization and analysis	SlicerIDCBrowser	https://github.com/ImagingDataCommons/SlicerIDCBrowser

Default Choice: idc-index is recommended for the vast majority of tasks (no authentication, simple API, supports batch downloads).

9. Integration with Analysis Pipelines

Integrate IDC data into imaging analysis workflows:

Read downloaded DICOM files:

import pydicom
import os

# Read DICOM files from downloaded series
series_dir = "./data/rider/rider_pilot/RIDER-1007893286/CT_1.3.6.1..."

dicom_files = [os.path.join(series_dir, f) for f in os.listdir(series_dir)
               if f.endswith('.dcm')]

# Load first image
ds = pydicom.dcmread(dicom_files[0])
print(f"Patient ID: {ds.PatientID}")
print(f"Modality: {ds.Modality}")
print(f"Image shape: {ds.pixel_array.shape}")

Build 3D volume from CT series:

import pydicom
import numpy as np
from pathlib import Path

def load_ct_series(series_path):
    """Load CT series as 3D numpy array"""
    files = sorted(Path(series_path).glob('*.dcm'))
    slices = [pydicom.dcmread(str(f)) for f in files]

    # Sort by slice position
    slices.sort(key=lambda x: float(x.ImagePositionPatient[2]))

    # Stack into 3D array
    volume = np.stack([s.pixel_array for s in slices])

    return volume, slices[0]  # Return volume and first slice for metadata

volume, metadata = load_ct_series("./data/lung_ct/series_dir")
print(f"Volume shape: {volume.shape}")  # (z, y, x)

Integration with SimpleITK:

import SimpleITK as sitk
from pathlib import Path

# Read DICOM series
series_path = "./data/ct_series"
reader = sitk.ImageSeriesReader()
dicom_names = reader.GetGDCMSeriesFileNames(series_path)
reader.SetFileNames(dicom_names)
image = reader.Execute()

# Apply processing (smoothing filter)
smoothed = sitk.CurvatureFlow(image1=image, timeStep=0.125, numberOfIterations=5)

# Save as NIfTI format
sitk.WriteImage(smoothed, "processed_volume.nii.gz")

Common Use Cases

Use Case 1: Find and Download Lung CT Scans for Deep Learning

Goal: Build training dataset of lung CT scans from NLST collection

Steps:

from idc_index import IDCClient

client = IDCClient()

# 1. Query lung CT scans with specific criteria
query = """
SELECT
  PatientID,
  SeriesInstanceUID,
  SeriesDescription
FROM index
WHERE collection_id = 'nlst'
  AND Modality = 'CT'
  AND BodyPartExamined = 'CHEST'
  AND license_short_name = 'CC BY 4.0'
ORDER BY PatientID
LIMIT 100
"""

results = client.sql_query(query)
print(f"Found {len(results)} series from {results['PatientID'].nunique()} patients")

# 2. Download data organized by patient
client.download_from_selection(
    seriesInstanceUID=list(results['SeriesInstanceUID'].values),
    downloadDir="./training_data",
    dirTemplate="%collection_id/%PatientID/%SeriesInstanceUID"
)

# 3. Save manifest for reproducibility
results.to_csv('training_manifest.csv', index=False)

Use Case 2: Query Brain MRI by Manufacturer for Quality Research

Goal: Compare image quality across different MRI scanner manufacturers

Steps:

from idc_index import IDCClient
import pandas as pd

client = IDCClient()

# Query brain MRI grouped by manufacturer
query = """
SELECT
  Manufacturer,
  ManufacturerModelName,
  COUNT(DISTINCT SeriesInstanceUID) as num_series,
  COUNT(DISTINCT PatientID) as num_patients
FROM index
WHERE Modality = 'MR'
  AND BodyPartExamined LIKE '%BRAIN%'
GROUP BY Manufacturer, ManufacturerModelName
HAVING num_series >= 10
ORDER BY num_series DESC
"""

manufacturers = client.sql_query(query)
print(manufacturers)

# Download samples from each manufacturer for comparison
for _, row in manufacturers.head(3).iterrows():
    mfr = row['Manufacturer']
    model = row['ManufacturerModelName']

    query = f"""
    SELECT SeriesInstanceUID
    FROM index
    WHERE Manufacturer = '{mfr}'
      AND ManufacturerModelName = '{model}'
      AND Modality = 'MR'
      AND BodyPartExamined LIKE '%BRAIN%'
    LIMIT 5
    """

    series = client.sql_query(query)
    client.download_from_selection(
        seriesInstanceUID=list(series['SeriesInstanceUID'].values),
        downloadDir=f"./quality_study/{mfr.replace(' ', '_')}"
    )

Use Case 3: Preview Series Without Downloading

Goal: Preview imaging data before deciding to download

from idc_index import IDCClient
import webbrowser

client = IDCClient()

series_list = client.sql_query("""
    SELECT SeriesInstanceUID, PatientID, SeriesDescription
    FROM index
    WHERE collection_id = 'acrin_nsclc_fdg_pet' AND Modality = 'PT'
    LIMIT 10
""")

# Preview each series in browser
for _, row in series_list.iterrows():
    viewer_url = client.get_viewer_URL(seriesInstanceUID=row['SeriesInstanceUID'])
    print(f"Patient {row['PatientID']}: {row['SeriesDescription']}")
    print(f"  View at: {viewer_url}")
    # webbrowser.open(viewer_url)  # Uncomment to auto-open

For more visualization options, see IDC Portal Getting Started Guide or SlicerIDCBrowser for 3D Slicer integration.

Use Case 4: License-Sensitive Batch Download for Commercial Use

Goal: Only download CC-BY licensed data suitable for commercial use

Steps:

from idc_index import IDCClient

client = IDCClient()

# Query only CC BY licensed data (allows commercial use with attribution)
query = """
SELECT
  SeriesInstanceUID,
  collection_id,
  PatientID,
  Modality
FROM index
WHERE license_short_name LIKE 'CC BY%'
  AND license_short_name NOT LIKE '%NC%'
  AND Modality IN ('CT', 'MR')
  AND BodyPartExamined IN ('CHEST', 'BRAIN', 'ABDOMEN')
LIMIT 200
"""

cc_by_data = client.sql_query(query)

print(f"Found {len(cc_by_data)} CC BY licensed series")
print(f"Collections: {cc_by_data['collection_id'].unique()}")

# Download after license verification
client.download_from_selection(
    seriesInstanceUID=list(cc_by_data['SeriesInstanceUID'].values),
    downloadDir="./commercial_dataset",
    dirTemplate="%collection_id/%Modality/%PatientID/%SeriesInstanceUID"
)

# Save license information
cc_by_data.to_csv('commercial_dataset_manifest_CC-BY_ONLY.csv', index=False)

Best Practices

• Check license before use - Always query license_short_name field and comply with license terms (CC BY vs CC BY-NC).
• Generate citations for attribution - Use citations_from_selection() to get properly formatted citations from source_DOI values and include them in publications.
• Start with small queries - Use LIMIT clause when exploring to avoid long downloads and help understand data structure.
• Use mini-index for simple queries - Only use BigQuery when you need comprehensive metadata or complex JOINs.
• Organize downloads with dirTemplate - Use meaningful directory structures like %collection_id/%PatientID/%Modality.
• Cache query results - Save DataFrames as CSV files to avoid repeated queries and ensure reproducibility.
• Estimate size before downloading - Check collection sizes before downloading—some collections are TB in size!
• Save manifests - Always save query results with series UIDs for reproducibility and data provenance tracking.
• Read documentation - IDC data structure and metadata fields are documented at https://learn.canceridc.dev/.
• Use IDC forum - Search for answers at https://discourse.canceridc.dev/, or ask questions to IDC maintainers and users.

Troubleshooting

Problem: ModuleNotFoundError: No module named 'idc_index'

• Cause: idc-index package not installed.
• Solution: Install with pip install --upgrade idc-index.

Problem: Download fails with connection timeout

• Cause: Unstable network or too large download.
• Solution:
  • Download in small batches (e.g., 10-20 series at a time).
  • Check network connection.
  • Use dirTemplate to organize downloads by batch.
  • Implement retry logic with delays.

Problem: BigQuery quota exceeded or billing error

• Cause: BigQuery requires a GCP project with billing enabled.
• Solution: For simple queries, use idc-index mini-index (no cost), or see references/bigquery_guide.md for cost optimization tips.

Problem: Cannot find series UID or no data returned

• Cause: UID typo, data not in current IDC version, or wrong field name.
• Solution:
  • Check if data is in current IDC version (some older data may be deprecated).
  • Test query with LIMIT 5 first.
  • Verify field names against metadata schema documentation.

Problem: Downloaded DICOM files won't open

• Cause: Download corrupted or viewer incompatible.
• Solution:
  • Check DICOM object type (Modality and SOPClassUID attributes) — some object types require specialized tools.
  • Verify file integrity (check file sizes).
  • Validate with pydicom: pydicom.dcmread(file, force=True).
  • Try different DICOM viewers (3D Slicer, Horos, RadiAnt, QuPath).
  • Re-download the series.

Common SQL Query Patterns

Quick reference for common queries. Detailed contextual examples are in the "Core Capabilities" section above.

Discover Available Filter Values

# What modalities exist?
client.sql_query("SELECT DISTINCT Modality FROM index")

# What body parts for a specific modality?
client.sql_query("""
    SELECT DISTINCT BodyPartExamined, COUNT(*) as n
    FROM index WHERE Modality = 'CT' AND BodyPartExamined IS NOT NULL
    GROUP BY BodyPartExamined ORDER BY n DESC
""")

# What manufacturers for MR?
client.sql_query("""
    SELECT DISTINCT Manufacturer, COUNT(*) as n
    FROM index WHERE Modality = 'MR'
    GROUP BY Manufacturer ORDER BY n DESC
""")

Find Annotations and Segmentations

Note: Not all image-derived objects belong to analysis result collections. Some annotations are stored with the original images. Use DICOM Modality or SOPClassUID to find all derived objects regardless of their collection type.

# Find all segmentations and structure sets via DICOM Modality

# SEG = DICOM Segmentation, RTSTRUCT = Radiotherapy structure set
client.sql_query("""
    SELECT collection_id, Modality, COUNT(*) as series_count
    FROM index
    WHERE Modality IN ('SEG', 'RTSTRUCT')
    GROUP BY collection_id, Modality
    ORDER BY series_count DESC
""")

# Find segmentations for specific collection (including non-analysis result items)
client.sql_query("""
    SELECT SeriesInstanceUID, SeriesDescription, analysis_result_id
    FROM index
    WHERE collection_id = 'tcga_luad' AND Modality = 'SEG'
""")

# List analysis result collections (curated derived datasets)
client.fetch_index("analysis_results_index")
client.sql_query("""
    SELECT analysis_result_id, analysis_result_title, Collections, Modalities
    FROM analysis_results_index
""")

# Find analysis results for a specific source collection
client.sql_query("""
    SELECT analysis_result_id, analysis_result_title
    FROM analysis_results_index
    WHERE Collections LIKE '%tcga_luad%'
""")

# Use seg_index for detailed DICOM segmentation metadata
client.fetch_index("seg_index")

# Get segmentation statistics by algorithm
client.sql_query("""
    SELECT AlgorithmName, AlgorithmType, COUNT(*) as seg_count
    FROM seg_index
    WHERE AlgorithmName IS NOT NULL
    GROUP BY AlgorithmName, AlgorithmType
    ORDER BY seg_count DESC
    LIMIT 10
""")

# Find segmentations for specific source images (e.g., chest CT)
client.sql_query("""
    SELECT
        s.SeriesInstanceUID as seg_series,
        s.AlgorithmName,
        s.total_segments,
        s.segmented_SeriesInstanceUID as source_series
    FROM seg_index s
    JOIN index src ON s.segmented_SeriesInstanceUID = src.SeriesInstanceUID
    WHERE src.Modality = 'CT' AND src.BodyPartExamined = 'CHEST'
    LIMIT 10
""")

# Find TotalSegmentator results in source image context
client.sql_query("""
    SELECT
        seg_info.collection_id,
        COUNT(DISTINCT s.SeriesInstanceUID) as seg_count,
        SUM(s.total_segments) as total_segments
    FROM seg_index s
    JOIN index seg_info ON s.SeriesInstanceUID = seg_info.SeriesInstanceUID
    WHERE s.AlgorithmName LIKE '%TotalSegmentator%'
    GROUP BY seg_info.collection_id
    ORDER BY seg_count DESC
""")

Query Slide Microscopy Data

# sm_index contains detailed metadata; join with index to get collection_id
client.fetch_index("sm_index")
client.sql_query("""
    SELECT i.collection_id, COUNT(*) as slides,
           MIN(s.min_PixelSpacing_2sf) as min_resolution
    FROM sm_index s
    JOIN index i ON s.SeriesInstanceUID = i.SeriesInstanceUID
    GROUP BY i.collection_id
    ORDER BY slides DESC
""")

Estimate Download Size

# Total size for specific criteria
client.sql_query("""
    SELECT SUM(series_size_MB) as total_mb, COUNT(*) as series_count
    FROM index
    WHERE collection_id = 'nlst' AND Modality = 'CT'
""")

Link to Clinical Data

client.fetch_index("clinical_index")

# Find collections with clinical data and their corresponding tables
client.sql_query("""
    SELECT collection_id, table_name, COUNT(DISTINCT column_label) as columns
    FROM clinical_index
    GROUP BY collection_id, table_name
    ORDER BY collection_id
""")

For complete schemas including value mappings and patient cohort selection, see references/clinical_data_guide.md.

Related Skills

The following skills complement the IDC workflow for downstream analysis and visualization:

DICOM Processing

• pydicom - Read, write, and manipulate downloaded DICOM files. Used to extract pixel data, read metadata, anonymize, and convert formats. Fundamental for processing IDC radiology data (CT, MR, PET).

Pathology and Slide Microscopy

• histolab - Lightweight slide extraction and preprocessing for Whole Slide Images (WSI). Used for basic slide processing, tissue detection, and dataset preparation of IDC slide microscopy data.
• pathml - Full-featured computational pathology toolbox. Used for advanced WSI analysis, including multiplexed imaging, nucleus segmentation, and training machine learning models on pathology data downloaded from IDC.

Metadata Visualization

• matplotlib - Low-level plotting library with full customization. Used to create static charts summarizing IDC query results (modality bar charts, series count histograms, etc.).
• seaborn - Statistical visualization library with pandas integration. Used for quick exploration of IDC metadata distributions, variable relationships, and categorical comparisons with beautiful default styles.
• plotly - Interactive visualization library. Used when you need hover info, zoom and pan capabilities to explore IDC metadata, or to create embeddable web dashboards for collection statistics.

Data Exploration

• exploratory-data-analysis - Comprehensive EDA for scientific data files. Used after downloading IDC data to understand file structure, quality, and characteristics before analysis.

Resources

Schema Reference (Primary Source)

Always use client.indices_overview to get current column schemas. This ensures consistency with your installed idc-index version:

# Get column names and types for any table
schema = client.indices_overview["index"]["schema"]
columns = [(c['name'], c['type'], c.get('description', '')) for c in schema['columns']]

Reference Documentation

• clinical_data_guide.md - Clinical/tabular data navigation, value mappings, and linking with imaging data.
• cloud_storage_guide.md - Direct access to cloud storage buckets (S3/GCS), file organization, CRDC UUID, versioning, and reproducibility.
• cli_guide.md - Complete idc-index command-line interface reference (idc download, idc download-from-manifest, idc download-from-selection).
• bigquery_guide.md - Advanced BigQuery usage guide for complex metadata queries.
• dicomweb_guide.md - DICOMweb endpoint URLs, code examples, and Google Healthcare API implementation details.
• indices_reference - External documentation for index tables (may be ahead of installed version).

External Links

• IDC Portal: https://portal.imaging.datacommons.cancer.gov/explore/
• Official Documentation: https://learn.canceridc.dev/
• Tutorials: https://github.com/ImagingDataCommons/IDC-Tutorials
• User Forum: https://discourse.canceridc.dev/
• idc-index GitHub: https://github.com/ImagingDataCommons/idc-index
• Citation: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180

Skill Updates

This skill version can be found in the skill metadata. To check for updates:

• Visit Releases page
• Watch the repository on GitHub (Watch → Custom → Releases)