flowchart LR A[(Database)] --> B["Data Retrieval<br/>(SQL/queries)"] B --> C["Result:<br/>records satisfying C"] A --> D["Data Mining<br/>(modeling)"] D --> E["Result:<br/>model/patterns"] E --> F["Action:<br/>decision, intervention,<br/>monitoring"]
19 Introduction to Data Mining and Process Frameworks
20 Introduction
Data mining is a formal discipline within the Knowledge Discovery in Databases (KDD) process. Its purpose goes beyond querying stored information to inferring structure: patterns, regularities, statistically significant relationships, and predictive models that capture latent behaviors in observational data. In technical terms, this involves articulating a coherent chain that spans from problem definition to deployment and monitoring of the results, with explicit validation and feedback stages.
In modern developments in the field, the growth in data volume, variety, and velocity necessitated the standardization of practices. This gave rise to process frameworks that act as end-to-end guides for conducting analytics and data mining projects consistently.
21 Data Retrieval versus Data Mining
Data retrieval answers questions fully defined by the user prior to accessing the system. Operationally, it is a transformation of the form: “give me the records satisfying condition C,” where C is known and the result is a subset of the repository.
Data mining, by contrast, seeks to answer questions where the structure of the relationship is unknown. The objective is not to filter, but to learn: to estimate functions, rules, or structures that explain or anticipate a phenomenon. For example, instead of “listing churned customers,” the question becomes “which variables anticipate churn and with what reliability?”, requiring variable definition, training set construction, bias control, evaluation metric selection, and uncertainty quantification.
22 Correlation and Causality
In data mining, a frequent finding is the presence of strong associations between variables. However, a statistical association (correlation) does not guarantee a causal link. The technical reason is that observational data can be dominated by omitted variables, non-random selection, confounders, or feedback loops between variables.
To establish causality, an experimental design (ideally randomized) is typically required. A classic example in digital systems is an A/B test: an intervention is randomly assigned to a treatment group and compared against a control group under a pre-defined metric (churn, conversion, retention). In practice, even with A/B testing, it remains necessary to monitor for violations: unit interference (spillover), temporal shifts (seasonality), and measurement biases.
flowchart TB U["Unobserved confounders<br/>(e.g., socioeconomic context)"] --> X["Variable X<br/>(academic GPA)"] U --> Y["Variable Y<br/>(income)"] X --> Y note1["If you observe X↔Y,<br/>do not conclude X→Y<br/>without controlling for U."] Y --> note1
23 Core Data Mining Tasks
23.1 Classification (supervised learning)
Classification models a categorical target variable. Given an attribute vector (^p), a function (f(){1,,K}) is learned that minimizes an expected loss (e.g., cross-entropy or approximate 0-1 loss). From a technical standpoint, performance is validated using schemes such as train/validation/test or k-fold cross-validation, and analyzed for sensitivity to class imbalance, decision thresholds, and asymmetric costs.
Example: predicting whether a student will pass given attendance, assignments handed in, and study hours. A baseline model is logistic regression; a more complex model could be gradient boosting if strong non-linearities exist.
23.2 Regression (supervised learning)
Regression is used when the target variable is continuous/numerical. The problem consists of approximating a function (y g()) under a criterion such as MSE or MAE. Technically, model selection depends on the expected structure: linearity, heteroscedasticity, collinearity, and presence of outliers. Typical metrics include RMSE, MAE, and (R^2), but decision contexts often incorporate business metrics (e.g., tolerable error per segment).
Example: predicting housing prices given physical and geographic attributes; if complex interactions are present, a non-linear model (trees, ensembles, neural networks) is preferred.
23.3 Clustering (unsupervised learning)
Clustering seeks natural partitions without target labels. The prototypical case is (k)-means, which minimizes the sum of squared distances to centroids. Its use demands technical decisions: variable standardization, choice of (k), sensitivity to initialization, and validity of the assumption of approximately spherical clusters. For data with complex geometries, DBSCAN or mixture models may be preferable.
Example: segmenting users by sessions per week, average session duration, and monthly spend. Cluster interpretation is not “in the data”; it is constructed by linking profiles with actionable interventions (campaigns, product features, support).
23.4 Association Rules
Rules of the form “if A then B” formalize co-occurrences (market basket analysis). The key metrics are support, confidence, and lift; lift is crucial because it avoids confusing high confidence with a highly frequent consequent.
Example: identifying that customers who buy cough syrup tend to buy throat lozenges; this enables cross-selling or joint product placement.
23.5 Anomaly Detection
Detects rare observations with respect to a normal pattern. Technically, it can be approached via distance to neighborhoods (LOF), isolation via partitions (Isolation Forest), reconstruction error (autoencoders), or probabilistic models. A critical point is that “rare” does not always mean “fraud”: it requires operational validation and a process for managing false positives.
Example: transactions of atypical amounts at unusual hours from distant locations; an anomaly score is assigned, and a review threshold is defined.
24 End-to-End Process and Data Mining Frameworks
As the discipline grew, frameworks emerged to organize projects in a repeatable manner. Here three traditional frameworks are integrated: KDD, CRISP-DM, and SEMMA, along with TDSP as a modern engineering-oriented alternative.
24.1 KDD as a Discovery Model
KDD describes the complete flow from raw data to knowledge: selection, preprocessing, transformation, mining, and evaluation/interpretation, culminating in actionable knowledge. It is especially useful when the objective is to understand and extract explainable patterns, in addition to predictive modeling.
flowchart LR A[(Data)] --> B["Selection<br/>(target dataset)"] B --> C["Preprocessing<br/>(noise, missing values)"] C --> D["Transformation<br/>(scaling, PCA, features)"] D --> E["Data Mining<br/>(models/patterns)"] E --> F["Interpretation<br/>& Evaluation"] F --> G["Knowledge<br/>(action/reports)"]
24.2 CRISP-DM as an End-to-End Project Framework
CRISP-DM organizes the project into six iterative phases: business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Its strength lies in managing the full project lifecycle with an explicit emphasis on putting results into operation (production, monitoring, business impact).
flowchart TB A[Business Understanding] --> B[Data Understanding] B --> C[Data Preparation] C --> D[Modeling] D --> E[Evaluation] E --> F[Deployment] E --> B D --> C note(["Iterative: if evaluation fails,<br/>return to data/model."]) E --> note
Example (hospital, appointment no-shows): a business objective is established (“reduce no-show rate”), available variables are audited (history, schedules), data is cleaned and consolidated (phones, duplicates), a model or experiment is built (e.g., non-attendance probability prediction or A/B testing SMS reminders), impact is evaluated, and the model is deployed with continuous monitoring.
24.3 SEMMA as a Practical Pipeline for Iterative Modeling
SEMMA (Sample, Explore, Modify, Model, Assess) emphasizes the core modeling cycle: sampling, exploring, modifying (feature engineering/transformations), modeling, and assessing. It is particularly useful when a well-defined dataset already exists and the goal is to optimize model performance by rapidly comparing alternatives.
flowchart LR S[Sample] --> E[Explore] E --> M[Modify] M --> Mo[Model] Mo --> A[Assess] A --> E A --> M
24.4 TDSP as a Modern Engineering-Oriented Alternative
TDSP (Team Data Science Process) is an agile, team-oriented process combining management, reproducibility, and deployment with engineering practices (version control, standardized artifacts, automated deployment). Compared to CRISP-DM, it emphasizes more explicitly “how to build” in production environments (data pipelines, repositories, CI/CD, traceability), while maintaining an end-to-end scope.
25 Framework Comparison (with Examples)
| Framework | Primary Focus | When to use it? | What it least emphasizes | Typical Deliverable | Clear Example |
|---|---|---|---|---|---|
| CRISP-DM | End-to-end process oriented toward business and deployment | Align problem, data, model, and operations with business impact metrics | Specific algorithms/techniques | Model + deployment plan + monitoring | Reduce appointment no-shows: objective → data → model/experiment → deployment of reminders |
| SEMMA | Iterative modeling pipeline (performance-focused) | Data is ready and quick iterations are needed to optimize metrics | Business context and operational deployment | Model benchmark and evaluation report | Testing 3 models to predict final grade, selecting best by RMSE/MAE |
| KDD | Knowledge discovery and interpretable patterns | Extracting patterns/insights with validated interpretation | Project management and deployment | Patterns/rules + interpretation | Market basket analysis: “if buying A, usually buys B” rules for cross-selling |
| TDSP | End-to-end with focus on engineering and teamwork | Production system requirement (pipelines, CI/CD, reproducibility) | Theoretical discovery framework; more prescriptive on implementation | Pipelines + artifacts + versioned deployment | Risk scoring with reproducible pipeline: ingestion → features → training → registry → serving |
26 How They Combine (A Practical Strategy)
In real-world projects, it is not necessary to choose a single framework. A common strategy is to use CRISP-DM for overall project governance, apply SEMMA within the modeling phase to iterate efficiently, and rely on KDD logic if the goal includes extracting explainable patterns or interpretable rules.
flowchart TB A["CRISP-DM: project governance"] --> B["Modeling"] B --> C["SEMMA: rapid iteration<br/>(of features/models)"] B --> D["KDD: interpretable<br/>patterns/insights"] C --> E["Evaluation & selection"] D --> E E --> F["Deployment & monitoring"]
27 Ethical and Implementation Considerations
In data mining, models learn patterns from past data; if historical data is biased, the model can amplify those biases. For this reason, beyond predictive metrics, fairness evaluations (by group or segment), reasonable explainability (where appropriate), and privacy governance are required. In implementation, operational stability is also crucial: data drift, shifts in data capture processes, seasonal variation, and performance degradation due to unseen conditions.
A key technical point is that a model with high validation performance may fail in production if the underlying distribution changes. Therefore, deployment must include monitoring: input data statistics, error rates, calibration, and alerting; as well as protocols for retraining and auditing.
28 Conclusions
Data mining transforms data into knowledge using supervised tasks (classification, regression), unsupervised tasks (clustering), and discovery methods (association rules, anomaly detection). To build robust projects, process frameworks provide clear structure: CRISP-DM organizes the full lifecycle through deployment, SEMMA accelerates modeling iteration, and KDD strengthens pattern discovery and interpretation, while modern alternatives like TDSP focus on reproducibility and software engineering.