BDA-Project-Report

Traditional ML approaches usually rely on strongly featured engineering and statistical strategies. In this context, Multinomial Naive Bayes (MNB) is probably one of the first and simplest processing techniques. MNB is a theorem of Bayes that processes with conditional independence of the features or attributes given the class label or the class category. Even if this assumption mostly doesn't hold for large data sets of the real-world case, MNB's operational efficiency and great usefulness for tasks like spam filtering and sentiment analysis on small data sets have made it popular among users. These drawbacks, however, stem from the model's inability to take into account the feature interdependence.

Support Vector Machines (SVMs) are the other key latest development of conventional ML for text classification. The main feature of SVMs is the ability to identify a hyperplane that separates the data with the greatest margin, which is known as a large margin classifier, based on the dimensionality of the feature space. Through kernel tricks, SVMs get the upper hand in case of non-linear decision boundaries, thus they can be used with text data having drastic multi-patterns. SVMs have the advantage when it comes to the models produced by combining features represented by TF-IDF, for instance, in the domain of document categorization and sentiment analysis. Nevertheless, their large computational cost and too much parameter tuning make it impossible to work with large-scale datasets.

Finally, the Random Forest method, as well as XGBoost and boosting methods such as LightGBM, can perform well as well in text classification. The Random Forest algorithm uses a tree ensemble learning algorithm that, through the selection of the best split and the repeated combinations of trees, helps to generalize better than single decision trees; while the algorithms of boosting, named now XGBoost and LightGBM, work by improving weak classifiers in a sequential fashion so to reduce the error. On the other hand, document ranking and biomedical text classification were the areas these methods were primarily applied in the structured text data applications. Despite their advantages, these procedures heavily depend on the feature engineering and can be found tricky if the data is sequential and context-sensitive.

Despite the traditional ML techniques, deep learning (DL) methods have altered text classification by obtaining feature representations from the text itself instead of hand-crafting them. With the introduction of FastText by Facebook, a simple yet effective machine learning algorithm for text categorization has been developed. It reflects words as dense vectors (word embeddings) and averages them to create document representational vectors. Furthermore, subword modeling helps to tolerate spelling mistakes and morphological variants, which are frequent in the case of low-resource languages. The downside of FastText is its straightforward approach, which may be a drawback when it comes to complex relationships between words.

Recurrent Neural Networks (RNNs) and their variations, such as Long Short-Term Memory (LSTM) networks, have incorporated sequential modeling capabilities into text categorization. LSTMs somewhat mitigate the vanishing gradient problem existing in conventional RNNs and maintain long-term dependencies. Consequently, they are viable candidates for applications which call for context comprehension, such as sentiment analysis or question-answering tasks. On the other side, RNNs and LSTMs read texts one after another, which elaborates on performance and the cost of computation and the time of training for big data or long sequences.

CNNs, developed primarily for image processing, have also demonstrated their capacity of being employed for text classification. In this way, by treating sentences as one-dimensional sequences where local features are extracted – these features can be n-grams via convolutional filters. Thus, they are probably the best in this class where it's essential to consider the local context, i.e., short text classification or sentiment analysis. CNNs are more efficient computationally, achieving increased parallel processing, yet they may still struggle to encode hints of long-term text dependencies.

The development that caused the greatest disruption in the text classification concept was through Transformer-based models, especially BERT, which is the Bidirectional Encoder Representations from Transformers. BERT introduced a new type of language modeling and used the transformer architecture which is self-attentional mechanisms. Differently from the previous models, BERT unifies the initial context with opposite directions, which results in a more profound knowledge of the relations within a sentence. BERT proves to be the most efficient model for various NLP tasks, such as text classification, sentiment analysis, and Named Entity Recognition (NER), by virtue that it has been pretrained on a huge dataset. It has an enormous capacity for consecutive learning and retraining, so it has become one of the most applicable models. Nevertheless, the model's weight and the high requirements of computational loads prevent small organizations and low-lifting environments from having visible results.

While ML methods such as MNB and SVM provide computational simplicity and efficiency, they still hugely depend on feature engineering and therefore tend to lose semantics and contextuality. In contrast, the DL approach, in particular that of transformer-based models, like BERT, champions this but it requires enormous computation power and huge amount of data for training. Accordingly, the methodology employed is often tailored to the task characteristics, like size of data, computation resources, and level of the text complexity. The combination of classical ML techniques and the recent innovations in DL methods have greatly extended the research horizons of text analysis, making the text classification area more dynamic and captivating.

The first stage is illegal text detection, which acts as a filtering point. The goal is to explore the way of finding and classifying the texts with the sensitive or illegal information. This hybrid stage combines rules-based systems and machine learning for a better precision.

For texts classified as non-illegal by the previous stage, the system proceeds to evaluate their quality through the high-quality comment filtering process. This stage involves the use of a pretrained BERT model for token embedding generation and an Auto-Encoder for reconstruction-based quality assessment.

This comprehensive two-stage pipeline ensures robust filtering of illegal content while simultaneously identifying high-quality comments for further use. By combining rule-based methods, traditional deep learning (FastText), and advanced transformer-based techniques (BERT), the system achieves a balance of efficiency, accuracy, and scalability.

In our project, we conducted extensive experiments to evaluate the performance of Machine Learning (ML) and Deep Learning (DL) models for illegal text detection. The results demonstrate that Deep Learning methods significantly outperform traditional ML approaches, leading to the selection of a two-stage hybrid approach combining FastText and BERT. This strategy achieves a balance between computational efficiency and classification accuracy.

This section details the Auto-Encoder architecture used for filtering high-quality comments and provides an analysis of its experimental results, including training performance and reconstruction error evaluation.

Since the current system operates with an unsupervised model for high-quality comment filtering, it is challenging to directly verify its effectiveness without labeled data. This limitation creates a gap in understanding how well the Auto-Encoder distinguishes between high- and low-quality comments in practical use cases.

The Auto-Encoder, as it is shaped at present, saturates full-connected layers for both its encoder and decoder placement. While the framework is made of simple anatomy, there is a great chance of not embracing all the complex relationships and patterns existing in the texts, especially in cases like lengthy and very intricate formats. The following improvements are proposed:

To validate the proposed improvements, the following steps can be undertaken:

Consequently, enhancement measures will strengthen the system to enable it to handle illegal text detection and filtering of quality comments effectively. The integrated form of modern architectures and real-time validation will surely eliminate existing boundaries and facilitate system capabilities.