Statistical Analysis and Data Mining The ASA Data Science Journal

Abstract High‐dimensional data in Euclidean space pose special challenges to data mining algorithms. These challenges are often indiscriminately subsumed under the term ‘curse of dimensionality’, more concrete aspects being the so‐called ‘distance concentration effect’, the presence of irrelevant attributes concealing relevant information, or simply efficiency issues. In about just the last few years, the task of unsupervised outlier detection has found new specialized solutions for tackling high‐dimensional data in Euclidean space. These approaches fall under mainly two categories, namely considering or not considering subspaces (subsets of attributes) for the definition of outliers. The former are specifically addressing the presence of irrelevant attributes, the latter do consider the presence of irrelevant attributes implicitly at best but are more concerned with general issues of efficiency and effectiveness. Nevertheless, both types of specialized outlier detection algorithms tackle challenges specific to high‐dimensional data. In this survey article, we discuss some important aspects of the ‘curse of dimensionality’ in detail and survey specialized algorithms for outlier detection from both categories. © 2012 Wiley Periodicals, Inc. Statistical Analysis and Data Mining, 2012

Abstract It is common to split a dataset into training and testing sets before fitting a statistical or machine learning model. However, there is no clear guidance on how much data should be used for training and testing. In this article, we show that the optimal training/testing splitting ratio is , where is the number of parameters in a linear regression model that explains the data well.

Random forest (RF) missing data algorithms are an attractive approach for imputing missing data. They have the desirable properties of being able to handle mixed types of missing data, they are adaptive to interactions and nonlinearity, and they have the potential to scale to big data settings. Currently there are many different RF imputation algorithms, but relatively little guidance about their efficacy. Using a large, diverse collection of data sets, imputation performance of various RF algorithms was assessed under different missing data mechanisms. Algorithms included proximity imputation, on the fly imputation, and imputation utilizing multivariate unsupervised and supervised splitting-the latter class representing a generalization of a new promising imputation algorithm called missForest. Our findings reveal RF imputation to be generally robust with performance improving with increasing correlation. Performance was good under moderate to high missingness, and even (in certain cases) when data was missing not at random.

How do we go about harnessing the “Big Data” paradigm?

Abstract Many real‐world networks are intimately organized according to a community structure. Much research effort has been devoted to develop methods and algorithms that can efficiently highlight this hidden structure of a network, yielding a vast literature on what is called today community detection . Since network representation can be very complex and can contain different variants in the traditional graph model, each algorithm in the literature focuses on some of these properties and establishes, explicitly or implicitly, its own definition of community. According to this definition, each proposed algorithm then extracts the communities, which typically reflect only part of the features of real communities. The aim of this survey is to provide a ‘user manual’ for the community discovery problem. Given a meta definition of what a community in a social network is, our aim is to organize the main categories of community discovery methods based on the definition of community they adopt. Given a desired definition of community and the features of a problem (size of network, direction of edges, multidimensionality, and so on) this review paper is designed to provide a set of approaches that researchers could focus on. The proposed classification of community discovery methods is also useful for putting into perspective the many open directions for further research. © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 4: 512–546, 2011

Abstract Many different relative clustering validity criteria exist that are very useful in practice as quantitative measures for evaluating the quality of data partitions, and new criteria have still been proposed from time to time. These criteria are endowed with particular features that may make each of them able to outperform others in specific classes of problems. In addition, they may have completely different computational requirements. Then, it is a hard task for the user to choose a specific criterion when he or she faces such a variety of possibilities. For this reason, a relevant issue within the field of clustering analysis consists of comparing the performances of existing validity criteria and, eventually, that of a new criterion to be proposed. In spite of this, the comparison paradigm traditionally adopted in the literature is subject to some conceptual limitations. The present paper describes an alternative, possibly complementary methodology for comparing clustering validity criteria and uses it to make an extensive comparison of the performances of 40 criteria over a collection of 962,928 partitions derived from five well‐known clustering algorithms and 1080 different data sets of a given class of interest. A detailed review of the relative criteria under investigation is also provided that includes an original comparative asymptotic analysis of their computational complexities. This work is intended to be a complement of the classic study reported in 1985 by Milligan and Cooper as well as a thorough extension of a preliminary paper by the authors themselves. Copyright © 2010 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 3: 209‐235, 2010

Abstract The class imbalance problem appears in many real‐world applications of classification learning. We propose an ensemble algorithm “Roughly Balanced (RB) Bagging” using a novel sampling technique to improve the original bagging algorithm for data sets with skewed class distributions. For this sampling method, the number of samples in the largest and smallest classes are different, but they are effectively balanced when averaged over all of the subsets, which supports the approach of bagging in a more appropriate way. Individual models in RB Bagging tend to show larger diversity, which is one of the keys of ensemble models, compared with existing bagging‐based methods for imbalanced data that use exactly the same number of majority and minority examples for every training subset. In addition, the proposed method makes full use of all of the minority examples by under‐sampling, which is efficiently done by using negative binomial distributions. Numerical experiments using benchmark and real‐world data sets demonstrate that RB Bagging shows better performance than the existing “balanced” methods and other common methods for area under the ROC curve (AUC), which is a widely used metric in the class imbalance problem. Copyright © 2009 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 2: 412‐426, 2009

Abstract Minimal depth is a dimensionless order statistic that measures the predictiveness of a variable in a survival tree. It can be used to select variables in high‐dimensional problems using Random Survival Forests (RSF), a new extension of Breiman's Random Forests (RF) to survival settings. We review this methodology and demonstrate its use in high‐dimensional survival problems using a public domain R‐language package randomSurvivalForest . We discuss effective ways to regularize forests and discuss how to properly tune the RF parameters ‘nodesize’ and ‘mtry’. We also introduce new graphical ways of using minimal depth for exploring variable relationships. © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 4: 115–132 2011

Abstract Change‐point detection is the problem of discovering time points at which properties of time‐series data change. This covers a broad range of real‐world problems and has been actively discussed in the community of statistics and data mining. In this paper, we present a novel nonparametric approach to detecting the change of probability distributions of sequence data. Our key idea is to estimate the ratio of probability densities, not the probability densities themselves. This formulation allows us to avoid nonparametric density estimation, which is known to be a difficult problem. We provide a change‐point detection algorithm based on direct density‐ratio estimation that can be computed very efficiently in an online manner. The usefulness of the proposed method is demonstrated through experiments using artificial and real‐world datasets. © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 2011

Abstract Seriation is an exploratory combinatorial data analysis technique to reorder objects into a sequence along a one‐dimensional continuum so that it best reveals regularity and patterning among the whole series. Unsupervised learning, using seriation and matrix reordering, allows pattern discovery simultaneously at three information levels: local fragments of relationships, sets of organized local fragments of relationships, and an overall structural pattern. This paper presents an historical overview of seriation and matrix reordering methods, several applications from the following disciplines are included in the retrospective review: archaeology and anthropology; cartography, graphics, and information visualization; sociology and sociometry; psychology and psychometry; ecology; biology and bioinformatics; cellular manufacturing; and operations research. Copyright © 2010 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 3: 70‐91, 2010

ABSTRACT We study covariance estimation in high‐dimensional approximate factor models with a sparse residual covariance. The estimator combines a log‐det loss with nuclear‐norm and off‐diagonal penalties on a sample‐normalized covariance scale. Under light‐tailed sampling, pervasive factors and factor‐space incoherence, we establish local curvature and characterize a local fixed point of the associated threshold iteration. Its low‐rank component consistently recovers the latent rank; exact signed residual‐support recovery additionally requires a minimum‐signal condition. We derive componentwise estimation rates and sufficient conditions for positive definiteness. The analysis separates the spectral requirements for rank recovery from the entrywise requirements for residual‐support recovery. A boundary‐adaptive search identifies the two thresholds by extending an initial grid when its selected solution lies on a boundary. Simulations examine structural recovery and covariance accuracy, and compare log‐det and Frobenius losses with covariance shrinkage methods. An application to European bank supervisory indicators illustrates the resulting decomposition into latent factors and sparse residual relationships.

ABSTRACT This paper addresses the fundamental challenge of jointly performing node clustering and representation learning in directed and valued graphs, which needs both global and local network structures to be captured. While these two tasks are highly interdependent, they are often treated separately in existing works. We propose the deep zero‐inflated latent position block model (Deep‐ZLPBM) in the context of directed and valued networks characterized by non‐symmetric adjacency matrices with positive integer entries. Our approach leverages a variational autoencoder (VAE) framework, combining a deep neural network encoder designed to handle directed edges and a zero‐inflated Poisson (ZIP) block modeling decoder to model sparse, integer‐weighted interactions. Recognizing the limitations of the standard evidence lower bound (ELBO) in VAEs, we explore the importance weighted ELBO (iw‐ELBO), a tighter bound on the marginal log‐likelihood optimized via gradient ascent, to enhance inference. Extensive experiments on synthetic datasets demonstrate that iw‐ELBO optimization yields significant performance gains. Moreover, our results validate that Deep‐ZLPBM effectively models complex network structures, providing interpretable partial memberships and insightful visualizations for directed, valued graphs.

ABSTRACT The aim of this study is to assess the impact of different dimensionality reduction techniques—specifically Principal Component Analysis (PCA) and Fisher's Linear Discriminant Analysis (LDA)—on the classification performance of Gradient Boosting Machines (GBM). While PCA is commonly employed to reduce feature dimensionality, often to speed up training or lower memory usage, its application is not standard in boosting pipelines. Moreover, to the best of our knowledge, no previous work has systematically explored whether LDA can enhance performance or is compatible with gradient boosting frameworks. In addition to evaluating the effects of PCA and LDA on the predictive performance of GBM, we introduce a novel integrated iterative algorithm that incorporates LDA‐based feature transformation at each boosting step. Our analysis spans five classical benchmark datasets, representing diverse scenarios in terms of sample size, number of features, for both binary and multiclass targets. Results show that applying LDA to extract features—and especially using the proposed LDA‐integrated GBM—improves classification accuracy compared to standard GBM, particularly in multiclass settings or in binary problems with high feature dimensionality. Finally, through a set of controlled simulations, we investigate how the performance of the integrated approach evolves with increasing feature dimensionality and varying levels of feature correlation and sample size.

ABSTRACT Modern biomedical and epidemiological studies increasingly combine data from multiple institutions, where differences in study populations and designs can induce cross‐source heterogeneity. To address this issue, we propose an integrative analysis framework for multisource partly interval‐censored data. The proposed method adopts a decomposition strategy to share information across studies while distinguishing common homogeneous effects from source‐specific heterogeneous effects. It also incorporates an MIC‐based ‐penalized procedure for tuning‐free variable selection, thereby enhancing selection stability and computational efficiency. In particular, this MIC‐based implementation avoids the joint tuning parameter search required by conventional penalties in the proposed two‐penalty framework. The estimation consistency, selection consistency, and asymptotic normality of the proposed estimators are established. Simulation studies and an application to the Alzheimer's Disease Neuroimaging Initiative and the National Alzheimer's Coordinating Center databases demonstrate the effectiveness and practical utility of the proposed method.

ABSTRACT This paper aims to develop a robust nonlinear model utilizing neural networks for function‐on‐function regression problems. The proposed framework is designed to enhance robustness in regression modeling by minimizing the impact of outliers in the response variable during model training. To achieve this, we introduce a robust loss function tailored to handle outliers effectively applied within a neural network based on basis expansion, and propose a novel functional normalization technique that normalizes the input function values of hidden layers with respect to time points. This approach demonstrates faster convergence as compared to existing neural network models and is less sensitive to hyperparameter settings. It also achieves high performance regression results even with outliers in the response variable. The effectiveness and superiority of the proposed method are validated through extensive simulation studies and real‐world applications using COVID‐19 excess mortality data.

ABSTRACT High Utility Sequential Pattern Mining (HUSPM) can evaluate the practical value of patterns while capturing temporal relationships. However, negative utility items (such as discounts, returns, and wastage) prevalent in real‐world applications pose two major challenges to existing methods: Firstly, negative utility items complicate the construction of projection databases and utility calculations. Traditional structures often require multiple accesses to the original sequence, incurring high time and memory costs; secondly, existing upper bounds often prove insufficiently compact when both positive and negative utilities coexist, thereby diminishing pruning effectiveness and inducing candidate explosion. To address these issues, this paper proposes the Rapid mining High Utility Sequential Pattern with Negative items (RHUSN) algorithm. This algorithm designs a compact data structure, Neg‐Seq‐Pro, which employs a shared negative sequence array and explicit projection lists to achieve unified modeling and rapid computation of both positive and negative utilities. Concurrently, it introduces a tighter utility upper bound, NTRSU, alongside a corresponding two‐tier pruning strategy comprising ‘prefix‐level pruning’ and ‘item‐level pruning’. This approach significantly reduces candidate expansion and redundant computation while ensuring result completeness. Extensive experimental results demonstrate that RHUSN achieves notable advantages over existing representative methods in terms of runtime, memory consumption, and the number of candidate patterns. This validates the effectiveness and scalability of the proposed data structure and upper bound pruning strategy in negative utility scenarios.

ABSTRACT Handling interval‐censored data poses significant challenges, particularly in multi‐state models where transition times are uncertain and computational complexity is high. This study investigates the use of landmark‐based estimators tailored for interval‐censored settings, with a particular focus on three‐state illness–death models. We propose two estimators: the Landmark‐Turnbull (LMTB) estimator, which integrates the Turnbull method for nonparametric survival estimation, and the Interval Landmark (ILM) estimator, a simplified alternative that uses midpoint imputation within the landmarking framework. Both approaches allow the estimation of transition probabilities without imposing the Markov assumption, thereby offering greater flexibility and interpretability in realistic clinical settings. The performance of the proposed estimators is compared with the standard Interval Aalen–Johansen (IAJ) estimator. Simulation studies and an application to synthetic data inspired by colon cancer patient trajectories demonstrate that LMTB yields more accurate estimates for long‐term transitions, while ILM is preferable for short‐term predictions due to its lower variance and computational simplicity.

ABSTRACT High‐dimensional tensor‐valued predictors are now routine in neuroimaging and other data‐rich domains, yet few statistical frameworks can jointly analyze continuous, binary, and count outcomes while scaling to realistic image resolutions. We propose the Bayesian sparse Kronecker product decomposition (BSKPD), which represents a regression or classification coefficient tensor as a low‐rank sum of Kronecker products of sparse component tensors. A sparse Kronecker product decomposition transformation reshapes tensor predictors and coefficients into lower‐dimensional matrices, enabling voxel‐level computation through standard matrix operations while preserving spatial structure. Sparsity is induced by a three‐parameter Beta–Normal (TPBN) global–local shrinkage prior on the Kronecker factors, yielding parsimonious and interpretable coefficient tensors that highlight informative brain regions. A unified exponential‐family formulation accommodates Gaussian, Bernoulli, and negative‐binomial responses, and Pólya–Gamma augmentation leads to closed‐form Gibbs updates. We establish identifiability and posterior consistency in both classical and high‐dimensional regimes, extending Bayesian theory to mixed‐type multivariate tensor regression. Simulations and Alzheimer's disease neuroimaging applications show that BSKPD yields interpretable whole‐brain coefficient maps while maintaining competitive predictive accuracy relative to existing approaches.

ABSTRACT We study the causal effects of urban amenities on nightly prices in short‐term rental platforms such as Airbnb. Prices are determined both by property characteristics and by their spatial location, which introduces complex local and neighborhood‐level variations. We analyze Airbnb listings in Broward County, Florida, using a two‐stage semiparametric framework with explicit spatial tuning. A generalized additive model captures nonlinear effects of property attributes and broad geographic trends, while a low‐rank Moran eigenvector basis filters residual fine‐scale spatial dependence. This approach provides a data‐driven framework to assess the spatially heterogeneous influence of amenities and neighborhood features, and offers interpretable insights into pricing strategies in densely distributed urban rental markets.

ABSTRACT Conformity of observed data with a postulated probability distribution is a fundamental problem of statistical modeling. Current tools for checking distributional conformity are often limited to univariate settings, with extensions to multivariate distributions facing challenges in interpretability and power. We propose a novel topological data analysis (TDA) framework that uses level sets to test distributional conformity, extending from univariate to bivariate applications. Level sets provide a natural geometric representation of probability distributions, capturing essential shape characteristics across dimensions. For univariate data, we construct level sets and compare them using the Dice Similarity Coefficient (DSC) to build a measure of distributional conformity; we then extend this methodology to bivariate settings through kernel density estimation and an annular decomposition of probability‐based level sets, with inference via an adaptive permutation test. Across simulation studies, the method maintains nominal Type I error control and achieves power competitive with established tests, including the Shapiro–Wilk test in the univariate case and the energy‐based E‐statistic and Henze–Zirkler test in the bivariate case, though no single method dominates uniformly across all alternatives. Rather than offering uniformly greater power, the framework is particularly well suited to settings where the goal is to characterize where and how distributions differ: the level set decomposition identifies which density regions drive a detected difference, and the average DSC provides a bounded, interpretable effect size on the unit interval. We illustrate these capabilities through applications to NHANES data, comparing cholesterol distributions and the joint distribution of systolic blood pressure and body mass index across age groups. This geometric interpretability, combined with natural extensibility to higher dimensions, positions our method as a valuable complement to existing approaches in the modern statistical toolkit.