PROJECT 03 / CLOUD & DATA ENGINEERING
Google Cloud
Dataproc Migration
Moving a notebook-based email classifier into managed Google Cloud compute, and learning what it takes to make a Spark and TensorFlow workflow run reliably.
The engineering problem
Our six-person CSC 595 team began with a Google Colab exercise for classifying labeled email messages as spam or ham. The migration objective was to run the data preparation and model workflow on Google Cloud Dataproc using two operating models: an on-demand Serverless Spark batch and a self-managed, single-node Dataproc cluster.
My documented hands-on work concentrated on cloud job configuration, dependency troubleshooting, PySpark-to-Keras workflow adaptation, output persistence and run validation. The project was a course exercise, not a production spam filtering service.
System architecture
Both deployment paths used the same core processing pattern. PySpark handled CSV ingestion, text cleanup, label normalization and class undersampling. The balanced records were then collected into the driver process with .toPandas() for scikit-learn splitting and TensorFlow/Keras model training. The neural network was not trained across distributed Spark workers.
Custom TensorFlow image
Pinned Python dependencies
The project also produced a numeric-only, BigQuery-compatible predictions CSV. Completed BigQuery imports or SQL queries were not verified.
Data processing and model
The 5,171-row input contained 3,672 ham and 1,499 spam messages. Spark parsed quoted and multiline CSV fields, normalized text and labels, removed punctuation and used seeded random undersampling of the majority ham class. Sampling was approximate rather than an exact fixed-size draw.
Prepare and balance
Filter to valid spam/ham labels, normalize text, sample ham without replacement, and shuffle with a reproducible seed.
Train in the driver
Collect balanced rows to pandas; use a stratified 80/20 split; fit the tokenizer on training text only; pad to 100 tokens. The Keras network used Embedding(32) → LSTM(16) → Dense(32, ReLU) → sigmoid.
Evaluate and persist
Threshold predictions at 0.5, calculate precision/recall/F1 and confusion counts, then save CSV outputs and model weights to Cloud Storage.
Two deployment approaches
| Aspect | Dataproc Serverless | Single-node cluster |
|---|---|---|
| Execution model | Submit batch; Google manages compute lifecycle | Create cluster, submit PySpark job; manual lifecycle |
| Successful run | April 20 and May 4, 2026 | April 21, 2026 |
| Verified region | us-east4 | us-central1 |
| TensorFlow setup | Custom container image | Configured Python dependencies |
| Model-quality metrics | May 4 run recovered | Not recovered |
Both modes completed a recorded PySpark job. However, the retained runtime observations came from different dates and setups. There is no supported speed, cloud-cost or model-quality winner between the two approaches.
Recorded Serverless evaluation
The archived May 4, 2026 driver log records a completed Serverless batch with 2,419 training rows, 605 evaluation rows and 10 training epochs. These are historical log values, not measurements from a currently accessible cloud environment.
F1: 0.9628. The four classification metrics are mathematically consistent with the confusion matrix below.
RECORDED CONFUSION MATRIX · 605 EVALUATION EMAILS
| Predicted ham | Predicted spam | |
|---|---|---|
| Actual ham | 284True negatives | 21False positives |
| Actual spam | 2False negatives | 298True positives |
These counts sum to 605. F1 = 2 × 298 / (2 × 298 + 21 + 2) = 0.9628 (rounded). The cloud cluster's corresponding classification metrics were not recovered.
Debugging and engineering decisions
Packaging TensorFlow for Spark jobs
An initial Serverless run failed because the standard environment did not contain TensorFlow. The workflow was adjusted to submit a Dataproc batch using a custom TensorFlow-capable container. The cluster deployment separately required explicit dependency configuration.
Persisting trained models to Cloud Storage
Saving a Keras model directly to a gs:// location encountered a libcurl/CA-certificate error. We saved the .keras file locally, then used Spark’s Hadoop FileSystem.copyFromLocalFile to transfer it to Cloud Storage. The later Serverless log records successful model and CSV output saves.
Keeping execution paths distinguishable
The cluster comparison script used separate cluster_spam_* artifact names. Original cluster job completion is supported by Dataproc/YARN status; the exact submitted script revision and cluster result CSVs are no longer available for independent verification.
Limitations and lessons
The model was trained on the Dataproc driver after .toPandas(), limiting dataset scalability. The class distribution was balanced before splitting, so the evaluation did not reflect natural incoming-email prevalence. The test partition was also reused for Keras validation; this weakens claims of independent generalization, particularly for the cluster variant with early stopping. Duplicate emails were present in the raw corpus, and duplicate-aware partition checking was not recorded.
The original Colab notebook and cloud versions also differed in preprocessing and training configuration, so comparing their separate accuracy figures would not establish a causal improvement from migration. This project did not verify production serving, BigQuery query execution, cloud cost savings or distributed neural-network training.
Team contribution and evidence
This was a six-person academic project. My retained Cloud Shell commands, debugging discussions and scripts support hands-on involvement in Dataproc submission, dependency troubleshooting, Spark/Keras adaptation, GCS artifact transfers and execution-state review. They do not establish sole authorship of the original notebook or every group script.
Source record: This case study was assembled from the archived CSC 595 presentation, Python scripts and a detailed project handoff auditing historical job logs. The original cloud resources are no longer accessible, and screenshots matching the successful model-evaluation run were not recovered. Scripts with incomplete revisions and account-specific configuration were intentionally not published as a runnable deployment package.
What this demonstrates
- Deploying PySpark jobs through two Dataproc operating models
- Configuring cloud Python dependencies and troubleshooting execution failures
- Separating distributed data preparation from driver-side model training
- Recovering a usable trained model and metrics through Cloud Storage artifact transfers
- Communicating experimental limitations and provenance of recorded results