SAFE: Scalable Automatic Feature Engineering Framework for Industrial Tasks

Nowadays, machine learning (ML) techniques have been widely explored and applied in almost all Internet companies, and serving as essential parts in diversified fields, such as recommendation system [4, 22, 9], fraud detection [2, 31, 33], advertising [15, 25, 34], and face recognition [1, 29], etc. With the help of these techniques, excellent performance and significant improvement have been obtained.

Generally speaking, to build a machine learning system, a professional and complex ML pipeline is always needed, which usually includes data preparation, feature engineering, model generation, and model evaluation, etc. It is widely agreed that the performance of machine learning methods depends to a large extent on the quality of the features, and generating a good feature set becomes a crucial step to chase high performance [7]. Therefore, most machine learning engineers take a large effort to obtain useful features when building a machine learning system.

However, it is frustrating that feature engineering is often the most indispensable part of human intervention in ML pipelines since human intuition and experience are gravely required, thus, it becomes tedious, task-specific and challenging, and hence, time-consuming. On the other hand, with the growing need for ML techniques in industrial tasks, it becomes impracticable to manually perform feature engineering in all of these tasks. This promotes the birth of automatic feature engineering, which is an important topic of automatic machine learning (AutoML) [28, 10, 35, 8]. The development of automatic feature engineering can not only liberate machine learning engineers from the substantial and tedious process, but also power machine learning techniques to be applied in more and more applications.

For a regular supervised learning task, problem can be formulated as using training examples to find a function $F:X\to Y$, which is defined as returning the $y$ value that obtains the highest score: $F\left(x\right)=\arg \underset{y}{max}S(x,y)$, where $X$ is the input space, $Y$ is the output space and $S:X\times Y\to R$ is a scoring function. The goal of automatic feature engineering is to learn a feature representation $\Psi :X\to Z$, to construct a new feature representation $z$ from the original feature $x$, with which the performance of subsequent machine learning tools can be further improved as much as possible.

Several studies have been conducted on this topic. To name a few, some methods use reinforcement learning based strategy to perform automatic feature engineering [6, 14, 30]. These methods require many rounds of attempts and it is necessary to generate a new feature set and evaluate it in each round, making them infeasible in industrial tasks. Transfer learning or meta-learning based strategies are also proposed for automatic feature engineering [12, 21]. However, a large number of experiments on various datasets are needed in advance to train these methods, and it is intractable to introduce new operators or increase the number of parent features. Some methods follow the generation-selection procedure [11, 16, 13] to do automated feature engineering. However, these methods always perform as generating all legal features in the feature generation stage and then selecting a subset features from them, thus the time and space complexity is extremely high, making it unapplicable for tasks with large data size or high feature dimension.

In industrial tasks, the size of real business data is always very huge, which introduces extremely high requirements for space and time complexity. At the same time, due to the rapid change of business, there are also high requirements for the flexibility and scalability of the algorithms. Besides, there are more requirements that need to be addressed [18]:

• Strong applicability: A tool that is highly adaptable means that it is user-friendly and easy to use. The performance of an automatic feature engineering algorithm should not depend on a large number of hyper-parameters or one of its hyper-parameter configurations can be applied to different data sets.

• Distributed computing: the number of samples and features in real-world business tasks are pretty large, which makes distributed computing necessary. Most parts of the automatic feature engineering algorithm should be able to be calculated in parallel.

• Real-time inference: real-time inference is involved in many real-world businesses. In such cases, once an instance is inputted, the feature should be produced instantly and the prediction can be performed subsequently.

In this paper, we approach the problem from the typical two-stage perspective and propose a method named SAFE (Scalable Automatic Feature Engineering) to perform efficient automatic feature engineering, which includes feature generation stage and feature selection stage. We guarantee computational efficiency, scalability and the requirements mentioned above. The major contributions of this paper are summarized as follows:

• In the feature generation stage, different from the previous methods which focuses on what operator to use or how to generate all legal features, we focus on mining the original feature pairs that generate more effective new features with higher probability, to improve the efficiency of the process.

• In the feature selection stage, we propose a pipeline of feature selection, with the consideration of the power of a single feature, the redundancy of feature pairs, and the feature importance evaluated by the typical tree model. It is suitable for multiple different business data sets and various machine learning algorithms.

• We have experimentally proved the advantages of our algorithm on a large set of data sets and multiple classifiers. Compared with the original feature space, the prediction accuracy is improved by $6.50\%$ on average.

The rest of this paper is organized as follows: Section II reviews the related work; Section III explains the problem setting; Section IV details the proposed method and provides some analyses; Section V states the detail of the data set, evaluation method and presents the experimental results to validate our method; Section VI concludes the paper.

As a nonnegligible issue for automatic machine learning [28, 10, 35, 8], automatic feature engineering has drawn extensive attention in recent years, and many methods have been proposed from different perspectives to solve this task [20, 6, 11, 12, 24, 5, 16, 13, 14, 30, 21]. In this section, we mainly discuss three typical strategies, which include the generation-selection strategy, reinforcement learning based strategy and transfer learning based strategy.

Given a supervised learning data set, a typical method for automatic feature engineering is to follow the generation-selection procedure. The FICUS algorithm [20] initializes by constructing a set of candidate features, and iterates to improve it until the computation budget is exhausted. During each iteration, it performs beam search to construct new features and selects features typically by using heuristic measures based on information gain in a decision tree. TFC [24] also solves this task by an iterative framework. In each iteration, it generates all legal features based on the current feature pool and all available operators, then selects the best features from all candidate features by using information gain, and keeps them as the new feature pool. With this framework, higher-order feature combinations can be obtained as the iteration progresses. However, the exhaustive search in each iteration leads to a combinatorial explosion of feature space, making this approach non-scalable. To avoid exhaustive search, learning based methods, such as the FCTree algorithm [5], have been proposed. FCTree trains a decision tree and performs feature generation by applying several sequential transformations to the original feature, and select features according to information gain on each node of the decision tree. Once a tree is built, features chosen at internal decision nodes are used to obtain the constructed features. [13] is a regression-based algorithm, which learns the representation by mining pairwise feature associations, identifying the linear or non-linear relationship between each pair, applying regression and selecting those relationships that are stable and improve the prediction performance. These algorithms always encounter performance and scalability bottlenecks since the cost of time and resource in the feature generation and selection procedure may be extremely unsatisfactory, if without ingenious design.

Reinforcement learning based strategies are also explored. [6] formalizes feature selection as a reinforcement learning problem and introduces an adaptation of the Monte-Carlo tree search. Here, the problem of choosing a subset of the available features is cast as a single-player game whose states are all possible subsets of features and the actions consist of choosing a feature and adding it to the subset. [14] handles this problem by exploring on a directed acyclic graph which represents the relationship between different transformed versions of the data, and learns an effective strategy to explore available feature engineering choices under a given budget through Q-learning. [30] formalizes this task as an optimization problem over a Heterogeneous Transformation Graph (HTG). It proposes a deep Q-learning on HTG to support efficient learning of fine-grained and generalized FE policies that can transfer knowledge of engineering “good” features from a collection of data sets to other unseen data sets.

Transfer learning or meta-learning based strategies are also proposed for automatic feature engineering. [12] employs learning to rank techniques to evaluate the newly constructed features and select the most promising ones. It is extremely time-consuming. For instance, their reported results were obtained after running for several days on moderately sized data sets. In contrast, [21] can generate effective features within seconds on average. It is based on learning the effectiveness of applying a transformation (e.g., arithmetic or aggregate operators) on numerical features, from past feature engineering experiences. However, since the meta-features do not take the relationship between features into account, it works better only when using unary transformations.

Beyond that, there are also other methods that direct at different settings. For example, [11] automatically constructs features from relational databases via deep feature synthesis. It focuses on the relationships between the various tables in the database to generate new features. A similar approach is adopted by [16]. What’s more, many studies try to perform feature engineering simultaneously while training the model, by introducing operations such as feature cross [27] or using techniques like self-attentive neural networks [26]. We need to address that, different from the methods that learn feature representations simultaneously with model training, we are aiming at learning a new representation for each sample based on the original features, which can be used to perform the subsequent machine learning models, and we have no constraint on what model to be used afterward. At the same time, for industrial tasks, interpretability is always required [32]. The generated features in our framework can be easily explained, to satisfy the interpretability requirement in industrial tasks.

To apply automatic feature engineering techniques in real-world applications, especially for industrial tasks, the efficiency and scalability of the aforementioned methods are still far from satisfactory. Methods with excellent efficiency and scalability, along with requisite interpretability and promising performance are in high demand.

Consider a predictive modeling task, which consists of:

• A dataset of input-output pairs. Let $x\in X$ be a record of the input space with $M$ original features $\left\{x^{\left(1\right)},\cdots ,x^{\left(M\right)}\right\}$. Let $y\in Y$ be the corresponding output label. Training data with $N$ records can be denoted as $D_{train}=\left\{X_{train},Y_{train}\right\}=\left\{(x_1,y_1),\cdots ,(x_N,y_N)\right\}$. Similarly, validation data and test data can be denoted as $D_{valid}$ and $D_{test}$.

• A machine learning algorithm $A$ that accepts a training set and a validation set as input and produces a function $F:X\to Y$, which return the predicted label $y$ given the input $x$.

• A loss function $L$ which computes the loss of a learned function $F$, according to the ground-truth label $y$.

The goal of automatic feature engineering is to learn a feature generation function $\Psi :X\to Z$ to generate new feature representation $z\in Z$ based on the original features $x\in X$, by using the set of $k$ operations $O=\left\{o_1,\cdots ,o_k\right\}$, so that the learning algorithm $A$ can find a function $F$ that minimizes the loss function $L$. i.e., to approximate the true underlying input-output function as much as possible. More formally, we want to obtain the feature generation function, with which the loss $L$ of the learned predictive function $F$ can be minimized:

in which the predictive function can be obtained by $F=A(D_{train\_new},D_{valid\_new})$, $D_{train\_new}$ and $D_{valid\_new}$ are the generated training and validation dataset, i.e., $D_{train\_new}=\left\{\Psi \left(X_{train}\right),Y_{train}\right\}$, and $D_{valid\_new}=\left\{\Psi \left(X_{valid}\right),Y_{valid}\right\}$.

The operators $O$, also known as $n$-ary operators, acts on $n$ original features for feature generation, and it can be divided into unary operators $O_1$, binary operators $O_2$, ternary operators $O_3$, etc. It should be noted that operators which do not satisfy the commutative property will be treated as multiple different operators in our subsequent descriptions and experiments, such as “$÷$”.

Unary operators are used for discretizing, normalizing, or mathematically transforming unit features:

• Discretization is the process of transferring continuous features into discrete features. It plays an important role in feature processing. It is robust to anomalous data and can make the trained model more stable. Typical feature discretization methods include ChiMerge, equidistant binning, equal-frequency binning, clustering binning, etc.

• Normalization refers to a process that makes features more normal or regular. Typical feature normalization methods include Min-Max normalization, Z-score, standardization of dispersion, etc.

• Mathematical transformations acting on unit features include log, sigmoid, square, square root, tanh, round, etc.

Binary operators combine two original features to generate a new feature:

• Four basic arithmetic operations: $+$, $-$, $\times$, $÷$.

• Logical operators act on two boolean features, such conjunction ($\wedge$), disjunction ($\vee$), alternative denial ($\uparrow$), joint denial ($\downarrow$), material conditional ($\to$), converse implication ($\leftarrow$), biconditional ($\leftrightarrow$), exclusive or ($\nleftrightarrow$), etc.

• GroupByThenMax, GroupByThenMin, GroupByThenAvg, GroupByThenStdev and GroupByThenCount. These operators implement the SQL-based operations with the same name.

• Ridge regression and kernel ridge regression in [13] can also be considered as binary operators.

Ternary operators combine three features to generate a new feature. A common ternary operator is a conditional operator, which is a basic conditional statement in many programming languages. For the conditional expression $a?b:c$, if the value of $a$ is true, the value of $b$ is obtained; otherwise, the value of $c$ is obtained.

There are also many operators that can accept multiple original features as input, such as MAX, MIN, MEAN, etc. We divide them into different categories when they accept a different number of inputs.

It should be pointed out that there are still many operators that apply in specific fields, we call them domain-specific operators, such as lag operators in time series analysis, genetic operators in biology, etc.

Because of the existence of various operators, an applicable automatic feature engineering algorithm framework should not limit operators and new operators should be easily added.

What’s more, to ensure the method to be feasible for large scale industrial tasks, the whole automatic feature engineering framework should be time and space-friendly, and with requisite interpretability and promising performance.

For simplicity and versatility, we only select four basic binary operators $+$, $-$, $\times$ and $÷$ when experimenting with each algorithm.