Objective
This project focuses on designing and implementing a deep generative model to create synthetic tabular data that can be used for privacy-preserving machine learning. By leveraging Python, deep learning frameworks, and generative AI techniques, you will build a Variational Autoencoder (VAE) geared towards generating realistic synthetic datasets. This approach ensures that sensitive information is kept confidential while maintaining the utility of the data for downstream tasks.
Learning Outcomes
- Understand the fundamentals and applications of generative adversarial models for synthetic data creation.
- Gain hands-on experience with Variational Autoencoders (VAEs) in real-world scenarios.
- Develop skills in data preprocessing and the evaluation of synthetic data quality.
- Learn how to balance model complexity with privacy constraints in data-driven projects.
Pre-requisite Skills
- Proficiency in Python programming
- Basic understanding of machine learning concepts
- Familiarity with deep learning frameworks (e.g., PyTorch or TensorFlow)
- Experience with data manipulation using libraries such as NumPy and Pandas
Skills Gained
- Implementing deep learning models with a focus on unsupervised learning
- Data preprocessing and visualization
- Training and fine-tuning deep generative models
- Evaluating synthetic data quality using statistical analyses and performance metrics
- Integrating generative AI techniques for privacy-aware ML solutions
Tools Explored
- Python
- PyTorch (or TensorFlow as an alternative)
- NumPy
- Pandas
- Matplotlib/Seaborn (for visualization)
- Scikit-learn (for metrics and additional data processing)
Steps and Tasks
Step 1: Data Collection and Preprocessing
Begin by selecting a suitable tabular dataset. For demonstration purposes, you could use a public dataset like the UCI Adult dataset. Load the data using Pandas, clean it, and transform any categorical variables into numerical representations.
import pandas as pd
from sklearn.preprocessing import LabelEncoder
# Load dataset
data = pd.read_csv("adult.csv")
# Drop missing values for simplicity
data = data.dropna()
# Encode categorical variables
for col in data.select_dtypes(include=['object']).columns:
le = LabelEncoder()
data[col] = le.fit_transform(data[col])
print("Dataset shape:", data.shape)
print(data.head())
Step 2: Building the Variational Autoencoder (VAE) Model
Design a VAE using PyTorch. The VAE will include an encoder to compress the data into a latent space and a decoder to reconstruct the data. This section details the architecture and core components.
import torch
import torch.nn as nn
import torch.nn.functional as F
class VAE(nn.Module):
def __init__(self, input_dim, hidden_dim, latent_dim):
super(VAE, self).__init__()
# Encoder layers
self.fc1 = nn.Linear(input_dim, hidden_dim)
self.fc_mu = nn.Linear(hidden_dim, latent_dim) # Mean vector
self.fc_logvar = nn.Linear(hidden_dim, latent_dim) # Log-variance vector
# Decoder layers
self.fc2 = nn.Linear(latent_dim, hidden_dim)
self.fc3 = nn.Linear(hidden_dim, input_dim)
def encode(self, x):
h = F.relu(self.fc1(x))
return self.fc_mu(h), self.fc_logvar(h)
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def decode(self, z):
h = F.relu(self.fc2(z))
return torch.sigmoid(self.fc3(h))
def forward(self, x):
mu, logvar = self.encode(x)
z = self.reparameterize(mu, logvar)
return self.decode(z), mu, logvar
# Example instantiation:
input_dim = data.shape[1]
hidden_dim = 64
latent_dim = 10
model = VAE(input_dim, hidden_dim, latent_dim)
print(model)
Step 3: Training the VAE Model
Train the model using an appropriate loss function combining reconstruction loss and the KL divergence. The training loop iteratively optimizes the network parameters.
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
# Convert data to PyTorch tensor
data_tensor = torch.tensor(data.values, dtype=torch.float32)
dataset = TensorDataset(data_tensor)
dataloader = DataLoader(dataset, batch_size=64, shuffle=True)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
def loss_function(recon_x, x, mu, logvar):
# Mean Squared Error reconstruction loss
recon_loss = F.mse_loss(recon_x, x, reduction='sum')
# KL divergence loss
KL_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon_loss + KL_loss
num_epochs = 50
model.train()
for epoch in range(num_epochs):
train_loss = 0
for batch in dataloader:
optimizer.zero_grad()
x_batch = batch[0]
recon_batch, mu, logvar = model(x_batch)
loss = loss_function(recon_batch, x_batch, mu, logvar)
loss.backward()
train_loss += loss.item()
optimizer.step()
print(f"Epoch {epoch+1}, Loss: {train_loss / len(dataset):.4f}")
Step 4: Generating Synthetic Data
After training the model, generate new synthetic data samples by sampling from the latent space and passing these samples through the decoder.
model.eval()
with torch.no_grad():
# Generate synthetic samples
num_samples = 1000
z = torch.randn(num_samples, latent_dim)
synthetic_data = model.decode(z).numpy()
# Convert to DataFrame for further inspection
synthetic_df = pd.DataFrame(synthetic_data, columns=data.columns)
print("Synthetic Data Sample:")
print(synthetic_df.head())
Step 5: Evaluating Synthetic Data Quality
Compare statistical properties (means, variances, and distribution shapes) of the original and synthetic datasets. Use visualization techniques to support your evaluation.
import matplotlib.pyplot as plt
import seaborn as sns
# Plot histograms for a few selected features
features = data.columns[:3] # Select first three features for demonstration
for feature in features:
plt.figure(figsize=(8,4))
sns.histplot(data[feature], color="blue", label="Original", kde=True, stat="density")
sns.histplot(synthetic_df[feature], color="orange", label="Synthetic", kde=True, stat="density")
plt.title(f"Distribution Comparison for {feature}")
plt.legend()
plt.show()
This comprehensive project not only enhances your understanding of generative models but also equips you with the vital skills needed for developing privacy-preserving synthetic data solutions in modern machine learning practices.