Predict Harmful Text

Introduction

In this mock interview, Jayaram (MLE @ Delivery Hero) is given a dataset of Twitter tweets and prompted to build a system that classifies harmful text.

Step 1: Understand the problem

To start, we’ll ask some clarifying questions:

1. Does the dataset have any features?
2. Does the dataset contain labels? This clarifies whether the ML task is a supervised or unsupervised.
3. Does the dataset contain more than 2 labels? This clarifies if this is a binary or multi-label classification problem.
4. How often should the predictions be computed?
5. Do we want one model per geography? This helps define the training overhead, data storage, and data privacy laws.
6. Will we use predictions for downstream tasks? If so, we may want to generate prediction probabilities.
7. Are we okay if the not-harmful text is marked harmful? This informs how we’ll select evaluation metrics (accuracy vs. precision).
8. Is there a model already in production? This informs how we’ll set up A/B testing, a roll out strategy, and key business metrics.

Step 2: Discuss the approach

We’ll build a text classifier through multiple components:

1. Text vectorization
2. Deep learning model
3. Inference component

Text vectorization

We’ll convert raw text to a sequence of integers/floats, which the ML algorithm will use to derive patterns and perform predictions. To do so, we’ll complete the following steps: Pre-process the text by removing punctuations and stop words, lemmatizing, and standardizing. Build a vocabulary, or map, that stores a word or token against an integer. Select a feature size for every token in the vocabulary, which captures contextual understanding of every token present in the corpus. Pad sequences with zeros so the training data is of equal shape.

All the above steps can be achieved by using the HuggingFace tokenizer. We’ll use the BERT Tokenizer (cased), based on subword tokenization.

Deep learning model

We’ll build a deep learning model using Tensorflow Keras Sequential API. The embedding layer will learn about tokens while training. This layer can also be exported and used as standalone in future tasks. The bidirectional long short-term memory (LSTM) layer is Tensorflow’s default option for GPU acceleration. The bidirectional layer allows us to pass information back and forth for the LSTM layer. The dense layers with ReLu activations will capture non-linearities. The final layer of size 1 will use sigmoid activation, to modify values to be between [0, 1]. We’ll use the binary cross entropy loss function and Adam optimizer.

Inference component

We’ll train on the training dataset, over 10 epochs and in batches of size 64. The train, validation, and test split will be split in a 70:20:10 ratio, respectively. Then, we’ll observe the train vs validation loss.

Classifier Class Signature

Python
class TextClassifier:
   def __init__(self, data: pd.DataFrame):
       self.data = data
       self.model = Sequential()
       self.vectoriser = TextVectorization()
       ...

   def preprocess(self, target_column: str) -> pd.DataFrame:
       pass

   def train(self, num_epochs: int, batch_size: int):
       pass

   def infer(self, text: str) -> str:
       pass

Step 3: Implement the algorithm

Python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding, Bidirectional
from tensorflow.keras.metrics import Precision, Recall, Accuracy

from transformers import BertTokenizer

class TextClassifier:

   def __init__(self, data: pd.DataFrame):
       self.data = data
       self.model = Sequential()
       self.tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
       self.vectorized_text = None

   def preprocess(self, target_column: str):
       self.X = self.data['tweet']
       self.y = self.data[target_column].values

       sequences = [sequence for sequence in self.X.values]
       model_inputs = self.tokenizer(sequences, padding=True, return_tensors="tf")
       self.vectorized_text = model_inputs["input_ids"]

   def _prepare_training_dataset(self, vectorized_text: pd.Series):
       dataset = tf.data.Dataset.from_tensor_slices((vectorized_text, self.y))
       dataset = dataset.cache()
       dataset = dataset.shuffle(160000)
       dataset = dataset.batch(16)
       dataset = dataset.prefetch(8) # helps bottlenecks

       train = dataset.take(int(len(dataset)*.7))
       val = dataset.skip(int(len(dataset)*.7)).take(int(len(dataset)*.2))
       test = dataset.skip(int(len(dataset)*.9)).take(int(len(dataset)*.1))

       return train, val, test

   def _build_model(self):
       # Create the embedding layer
       self.model.add(Embedding(len(self.tokeniser.get_vocab)+1, 32))
       # Bidirectional LSTM Layer
       self.model.add(Bidirectional(LSTM(32, activation='tanh')))
       # Feature extractor Fully connected layers
       self.model.add(Dense(128, activation='relu'))
       self.model.add(Dense(256, activation='relu'))
       self.model.add(Dense(128, activation='relu'))
       # Final layer
       self.model.add(Dense(1, activation='sigmoid'))
       self.model.compile(loss='binary_crossentropy', optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001), metrics = ["accuracy"])

   @property
   def get_model_summary(self):
       return self.model.summary()

   @property
   def get_training_history(self):
       return self.history

   def train(self, epochs: int = 10, batch: int = 16):
       train, val, test = self._prepare_training_dataset(vectorized_text=self.vectorized_text)
       self._build_model()
       # uncomment when using a GPU
       # with tf.device('/device:GPU:0'):
       self.history = self.model.fit(train, epochs=epochs, batch_size=batch, validation_data=val)

       precision, recall, accuracy = self.evaluate(test=test)
       print(f'Precision: {precision.result().numpy()},\
               Recall:{recall.result().numpy()},\
               Accuracy:{accuracy.result().numpy()}'
       )

   def evaluate(self, test):
       pre = Precision()
       rec = Recall()
       acc = Accuracy()

       for batch in test.as_numpy_iterator():
           # Unpack the batch
           X_true, y_true = batch
           # Make a prediction

           yhat = self.model.predict(X_true)

           # Flatten the predictions
           y_true = y_true.flatten()
           yhat = yhat.flatten()

           pre.update_state(y_true, yhat)
           rec.update_state(y_true, yhat)
           acc.update_state(y_true, yhat)
       return pre, rec, acc

# load the dataset
df = pd.read_csv('tweets_flagged.csv')

clf = TextClassifier(data=df)
clf.preprocess(target_column='harmful')
clf.train(epochs=10, batch=64)

clf.model.save(‘harmful_text_classsifier.h5’)

Step 4: Test code & discuss results

Python
# load the model
model=tf.keras.models.load_model('harmful_text_classsifier.h5')

vectorised_text = clf.tokenizer([text], padding=True, return_tensors="tf")['input_ids']

res = model.predict(vectorised_text)
if (res >= 0.5).astype(int):
  print(res, "Harmful")
else:
  print(res, "Not Harmful")

Now that we have the results, we’ll review the following:

1. Training vs. validation loss. We’ll see if the values converge over epochs, and if the loss and val_loss improves on every epoch.
2. Signs of overfitting and underfitting. In cases of overfitting, we can utilize techniques like regularization and batch normalization. In cases of underfitting, we might check if the features are sufficient, consider better text encoding techniques, or consider pre-trained embeddings (e.g. word2vec, glove).
3. Test runs to verify that predictions are accurate.

To improve the model, we can consider the following:

1. Enable the model to capture contextual information across positive and negative words.
2. Train the model on more epochs.
3. Improve the text vectorizer and pre-processing data pipeline.
4. Use transfer learning techniques.

Interview improvements

Some opportunities for improvement in the interview include:

1. Suggesting different approaches for the data science algorithm, instead of jumping directly to using neural networks and LSTMs. We could discuss the algorithm’s drawbacks in fitting to the current problem.
2. Justifying the chosen evaluation metrics with more clarity. We could have started with why accuracy is a best fit for this particular case, and what the downsides of accuracy are in cases of imbalanced data.
3. Explaining the attention mechanism more effectively. We could have started with what LSTMs lack, especially when learning from long sentences. For example, LSTMs focus on the end words, the information earlier in the sentence is missed. A better way to assess a sentence is to focus on the most important parts of the sentence by assigning weights.