Event Data Pipeline

Overview

This project is a real-time, event-driven data pipeline built to ingest, process, store, and visualize e-commerce behavioral data. It is designed around a streaming architecture where raw user events such as product views, cart additions, and purchases flow continuously through the system and are transformed into queryable analytics.

The pipeline uses Apache Kafka as the message broker to decouple data producers from consumers. A Spark cluster reads from Kafka, applies transformations and aggregations, and writes results to two destinations: MinIO, which serves as the raw data lake in S3-compatible object storage, and Elasticsearch, which indexes the processed data for fast search and retrieval. Kibana sits on top of Elasticsearch to provide dashboards for exploring customer behavior patterns, product trends, and purchase funnels.

All external-facing services are routed through an NGINX reverse proxy with TLS termination, so every web interface is served over HTTPS. Prometheus and Metricbeat handle metrics collection, with Kibana also serving as the observability frontend for security and health monitoring.

The entire stack runs in Docker containers orchestrated with Docker Compose, making it straightforward to spin up locally or adapt for a cloud environment.

Data flow: Kafka -> Spark -> MinIO (data lake) + Elasticsearch -> Kibana

---

System Requirements

• OS: Ubuntu 22.04 LTS (or WSL2 with Ubuntu)
• Memory: 8GB RAM minimum (16GB recommended)
• Storage: 20GB free disk space
• Software: Docker + Docker Compose
• Runtime: Java 11+ (OpenJDK recommended)

---

Data Setup

1. Download the dataset:

   • Visit: Kaggle - eCommerce Events History
   • Download the CSV file
   • Place it in the data/ folder as events.csv

2. Validate your data (optional):

   # Check if data file exists and preview structure
   ls -la data/
   head -5 data/events.csv
   
   # Verify CSV has required columns: event_time, event_type, product_id, price, user_id

---

Installation

Step 1: Install Docker

For WSL2/Ubuntu:

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker Desktop for Windows and enable WSL connection

Step 2: Configure System

# Set Elasticsearch memory limits (required)
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Step 3: Setup Project

# Install dependencies
sudo apt install python3 python3-pip python3-venv git curl openjdk-11-jdk openssl -y

# Create Python virtual environment
python3 -m venv venv
source venv/bin/activate

# Install required packages
pip install -r requirements.txt

# Generate TLS certificates and start services
./generate-certs.sh
docker compose up -d

# Wait for initialization (2-3 minutes)
sleep 120

# Check service status
docker compose ps

Usage

Data Ingestion

# Stream CSV data to Kafka topics
docker-compose --profile ingestion up ingestion-service

Data Processing

# Process streamed data (consumes from Kafka, processes with Spark)
docker-compose --profile processing up spark-processor

# Monitor system health
python monitoring/pipeline_monitor.py

Web Interfaces

Service	URL	Description
Kibana Dashboard	https://localhost/	Customer analytics and visualizations (main interface)
Spark Master UI	https://localhost/spark/	Monitor cluster and job execution
MinIO Console	http://localhost:9001	Data lake management (login: minioadmin/minioadmin123)
Elasticsearch API	https://localhost/elasticsearch/	Direct database queries

---

Service Management

# Start/stop all services
docker compose up -d
docker compose down

# View service logs
docker compose logs -f [service-name]

# Scale Spark workers
docker compose up -d --scale spark-worker=3

---

Service Ports

Service	Internal Port	External Access	Description
NGINX (HTTPS)	443	https://localhost	TLS reverse proxy (entry point)
Kibana Dashboard	5601	https://localhost/	Analytics dashboard (proxied via NGINX)
Spark Master	8080	https://localhost/spark/	Cluster monitoring (proxied via NGINX)
MinIO Console	9001	http://localhost:9001	Storage management (direct access)
Elasticsearch	9200	https://localhost/elasticsearch/	Search API (proxied via NGINX)
Prometheus	9090	http://localhost:9090	Metrics monitoring (direct access)
Kafka	9092	Internal only	Message streaming (docker network)
Zookeeper	2181	Internal only	Kafka coordination (docker network)

Quick access:

• Main dashboard: https://localhost/ (Kibana)
• Spark monitoring: https://localhost/spark/ (Spark Master UI)
• Storage management: http://localhost:9001 (MinIO Console)
• Metrics: http://localhost:9090 (Prometheus)
• Search API: https://localhost/elasticsearch/ (Elasticsearch)
• Security check: run ./security-check.sh in terminal

---

TLS Security

The pipeline uses NGINX as a reverse proxy with TLS termination to provide HTTPS across all web interfaces.

First-Time Setup

# Generate SSL certificates (included script)
./generate-certs.sh

# Start services with HTTPS enabled
docker compose up -d

Browser Security Warning

Self-signed certificates will trigger a browser warning. Click "Advanced" and then "Proceed to localhost" to continue. For production use, replace the self-signed certs with certificates from a trusted CA.

Security Monitoring

Prometheus and Metricbeat collect metrics from all services, which are surfaced through Kibana dashboards. A Python script handles log-based threat detection.

# Start monitoring services
docker compose up -d prometheus metricbeat

# Run a security status check
./security-check.sh

# Test threat detection rules
./test-security.sh

What gets monitored:

• HTTPS endpoint health — NGINX, Kibana, Spark UI accessibility
• Certificate security — TLS certificate strength and expiry tracking
• Threat detection — SQL injection, XSS, brute force, and path traversal attempts
• Access patterns — external IP monitoring and suspicious request patterns
• Authentication failures — failed login attempts across all services
• Service availability — MinIO Console, Prometheus, Elasticsearch health

View results:

• Kibana dashboards: https://localhost/
• Prometheus console: http://localhost:9090
• Security reports: run ./security-check.sh anytime

---

Troubleshooting

Services not starting:

docker compose logs -f elasticsearch
docker compose logs -f kafka

# Restart a specific service
docker compose restart elasticsearch

Out of memory errors:

free -h
docker stats

# Increase Elasticsearch heap size if needed
export ES_JAVA_OPTS="-Xms2g -Xmx2g"
docker compose up -d elasticsearch

Data file not found:

ls -la data/events.csv
head -5 data/events.csv

Python import errors:

source venv/bin/activate
pip install --upgrade -r requirements.txt

---