Part 0 — Data Preparation#

Goal of this notebook: load the raw multi-omics TCGA-BRCA data, split patients into train/test sets, and save that split to disk so every later notebook in the workshop uses the exact same patients for training and testing.

We work with three “omic views” for each patient:

View	What it measures	# features
`transcriptomics`	gene expression (RNA-seq)	~30,000
`proteomics`	protein abundance	~460
`methylation`	DNA methylation (epigenetic marks)	~200,000

The prediction target is the PAM50 molecular subtype of breast cancer (LumA, LumB, Basal, Her2, Normal).

1. Imports#

from pathlib import Path
import pickle

from s2_helpers import load_omics
from sklearn.model_selection import train_test_split

2. Load the multi-omics data#

load_omics reads the pre-processed TCGA-BRCA data for the requested omic views and returns:

X_views — a dict mapping each omic name to a (patients × features) DataFrame
y_raw — a Series of PAM50 subtype labels, indexed by patient ID

All views and the label are aligned on the same set of patients.

# Path to the shared (workshop-wide) data directory
DATA_DIR = Path("/data/")

# Load the three omic views we'll use throughout the workshop, plus the subtype labels
X_views, y_raw = load_omics(
    DATA_DIR,
    omic_keys=["transcriptomics", "proteomics", "methylation"],
)

Omic view dimensions:
  transcriptomics:  500 patients ×  29995 features
  proteomics     :  500 patients ×    464 features
  methylation    :  500 patients × 200000 features

Subtype counts:

paper_BRCA_Subtype_PAM50
LumA      237
LumB      100
Basal      97
Her2       41
Normal     25
Name: count, dtype: int64

3. Create a stratified train/test split#

We split on patient IDs only (not on any single omic matrix) so the same patients end up in the same split across every view. stratify=y_raw keeps the proportion of each PAM50 subtype roughly equal between train and test — important here since some subtypes (e.g. Normal, Her2) have relatively few patients.

RANDOM_STATE is fixed so the split is reproducible across the whole workshop.

RANDOM_STATE = 42

train_ids, test_ids = train_test_split(
    y_raw.index.to_numpy(),  # numpy array: sklearn can't index pandas 3's arrow-backed Index
    test_size=0.25,   # 75% train / 25% test
    random_state=RANDOM_STATE,
    stratify=y_raw,   # preserve subtype proportions in both splits
)

print(f"Training patients : {len(train_ids)}")
print(f"Test patients     : {len(test_ids)}")

Training patients : 375
Test patients     : 125

4. Save the split for later notebooks#

We persist the train/test patient IDs to a pickle file. Every subsequent notebook in the workshop loads this file instead of re-splitting, so results stay consistent and comparable across notebooks and participants.

splits_path = DATA_DIR / "patient_splits.pkl"

try:
    with open(splits_path, "wb") as f:
        pickle.dump({"train_ids": train_ids, "test_ids": test_ids}, f)
    print(f"Saved patient splits to: {splits_path}")
except PermissionError:
    # The shared workshop data dir is read-only. The split is deterministic
    # (fixed seed), so the pre-generated file there must match what we just
    # computed - verify that instead of writing.
    with open(splits_path, "rb") as f:
        existing = pickle.load(f)
    assert set(existing["train_ids"]) == set(train_ids), (
        f"{splits_path} does not match this run's split - "
        "ask an instructor to regenerate it from the current omics.pkl")
    print(f"Data dir is read-only; verified pre-generated splits at: {splits_path}")

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[5], line 3
      1 splits_path = DATA_DIR / "patient_splits.pkl"
      2 
----> 3 with open(splits_path, "wb") as f:
      4     pickle.dump({"train_ids": train_ids, "test_ids": test_ids}, f)
      5 
      6 print(f"Saved patient splits to: {splits_path}")

PermissionError: [Errno 13] Permission denied: '/data/patient_splits.pkl'