{"id":1073,"date":"2017-09-01T17:47:28","date_gmt":"2017-09-01T15:47:28","guid":{"rendered":"http:\/\/www.sarahmestiri.com\/?p=1073"},"modified":"2017-09-02T11:31:22","modified_gmt":"2017-09-02T09:31:22","slug":"applied-text-classification-on-email-spam-filtering-part-1","status":"publish","type":"post","link":"https:\/\/www.sarahmestiri.com\/index.php\/2017\/09\/01\/applied-text-classification-on-email-spam-filtering-part-1\/","title":{"rendered":"Applied text classification on Email Spam Filtering [part 1]"},"content":{"rendered":"

Since last few months, I’ve started working on online\u00a0Machine Learning Specialization<\/a> provided by the University of Washington. The first course was about ML foundations<\/em>, the second about Linear Regression<\/em>, and the third course on which \u00a0I’m currently on track is about Classification<\/em>. I liked the courses almost in every aspect as they\u00a0teach ML algorithms implementation from Scratch. That was my goal when I decided to discover the field in more depth. But, honestly, I was feeling that there is a gap somehow because many questions were left without answers along the way. Then, after reading about how to start with Machine Learning, I found that most articles emphasized the importance of combining courses <\/strong>with practical projects<\/strong> in order to apply what’s learned and better assimilate it.. and it’s so true! You just have to try to combine both and you will soon notice the difference!<\/p>\n

So, here I am in my first practice application! \ud83d\ude00 I chose to try Email Spam Filtering<\/em><\/strong> as it’s a very common topic in applied classification. It can be understood easily because we are experiencing spam filtering in our e-mails every day.<\/p>\n

I followed a simple starting tutorial: Email Spam Filtering: A Python implementation with Scikit-learn<\/a>. Soon after finishing it, my brain started analyzing the steps and a bunch of questions bombarded my head!<\/p>\n

Why “equal number of spam and non-spam emails”? What’s stemming? Are there other methods to clean data other than removing stop words and lemmatization? How is the partition between the training set and test set done? Why no validation set? Why Naive Bayes classifier and SVM (Support Vector Machines) were specifically used? What makes Naive Bayes so popular for document classification problem? etc..<\/p>\n

As William.S.Burroughs said, “Your mind will answer most questions if you learn to relax and wait for the answer.”<\/p><\/blockquote>\n

I took a breath and started answering question by question by doing sometimes search on the net, experimenting some changes in code and analyzing the output. And I’m happily sharing \u00a0the results:<\/p>\n

1) The data we need<\/span><\/h3>\n

– how many emails we\u2019ve seen (will be used in train-test sets)<\/span>
\n– how many emails go in each label (used to\u00a0detect if there is imbalanced data)<\/span>
\n– how often a word is associated with each label (used to calculate the probability of an email being a spam or ham (class 0 or class 1))<\/span><\/p>\n

2) Cleaning data<\/span><\/h3>\n

Why cleaning the words list?<\/strong> Cleaning data is essential in order to reduce the probability of getting wrong results because some words have no influence on the classification (they can neither be associated with spam class and nor\u00a0with ham class) and there are words that can be normalized in order to group same-meaning words and reduce redundancy.\u00a0By acting on the quality of the training data, we can change what is called the\u00a0accuracy<\/a>\u00a0of the classifier. So removing stop words, stemming<\/em>, and lemmatization\u00a0<\/em>help in improving the results of Machine Learning algorithms.<\/p>\n

3) Naive Bayes<\/span><\/h3>\n

Why was Naive Bayes used?<\/strong> Naive Bayes has\u00a0highly\u00a0effective learning and prediction,\u00a0it’s often used to compare with more sophisticated methods because it’s fast and highly scalable (works well with high-dimensional data) and as Andrew Ng suggests when dealing with an ML problem start by trying with a simple quick and dirty algorithm and then expand from that point.<\/span><\/p>\n

How is Naive Bayes simple and easy?\u00a0<\/strong>Naive Bayes is based on “Bayes” theorem and it’s called “Naive” because it\u00a0assumes that features are independent<\/em> of each other given the class(no\/little correlation between features), which is not realistic. Thus,\u00a0<\/span>Naive Bayes can learn individual features importance but can\u2019t determine the relationship among features. Besides, the training time with Naive Bayes is significantly smaller as opposed to alternative methods and it doesn’t require much training data.<\/span><\/p>\n

Why Multinomial Naive Bayes? What about other models like Gaussian Naive Bayes or Bernoulli Naive Bayes?<\/strong><\/p>\n

Well, Multinomial NB<\/em><\/strong> considers the frequency<\/em>\u00a0<\/strong>count<\/strong><\/em>\u00a0<\/strong>(occurrences)\u00a0of the features<\/em> (words in our case) while Bernoulli NB<\/strong> cares only about the presence or absence<\/strong> of a particular feature (word) in the document. The latter is adequate for features that are binary-valued\u00a0<\/em>(Bernoulli, boolean). \u00a0Whereas, with Gaussian NB<\/strong>, features are real-valued<\/em> or continuous\u00a0<\/em>and their distribution is Gaussian, the Iris Flower dataset is an example with continuous features.<\/p>\n

4) Support Vector Machines (SVM)<\/span><\/h3>\n

Why was SVM used?\u00a0<\/strong>I didn’t find a specific reason to that, but what I learned is that SVM delivers high accuracy results because it uses\u00a0an optimization procedure.\u00a0<\/span>SVM builds a classifier by searching for a separating hyperplane (<\/span>optimal hyperplane<\/b>) which is optimal and <\/span>maximises <\/b>the margin <\/b>that separates\u00a0the categories (in our case spam and ham). Thus, SVM has the advantage of robustness in general and effectiveness when\u00a0the number of dimensions is greater than the number of samples.<\/span><\/p>\n

Unlike Naive Bayes, SVM is a non-probabilistic algorithm.<\/span><\/p>\n

What’s\u00a0the difference between LinearSVC and SVC (Scikit-learn)?\u00a0<\/strong>The difference is that they don’t accept the same parameters. For example, LinearSVC does not accept kernel parameter as it’s supposed linear. SVC supports more parameters(C, gamma,..) since it holds all possible kernel functions<\/a> (linear, polynomial, rbf or radial basis function, sigmoid).<\/p>\n

How can tuning SVM parameters be done?\u00a0<\/strong>Tuning SVM parameters improve the performance of the algorithm. Some of them have a higher impact:<\/p>\n

-Kernel<\/strong>: Kernal is like a similarity function. It’s a way of computing the dot product of two vectors in possibly a high dimensional feature space using data transformations based on some provided constraints into a more complex space. Kernel functions are sometimes called “generalized dot product”.<\/p>\n

-Gamma:<\/strong>\u00a0Kernel coefficient for \u2018rbf\u2019, \u2018poly\u2019 and \u2018sigmoid\u2019. Higher the value of gamma will try to exactly fit as per training data set i.e. generalization error and cause an over-fitting problem.<\/p>\n

-C<\/strong>: The factor\u00a0C<\/i>\u00a0in (3.15) is a parameter that allows one to trade off training error vs. model complexity. A small value for\u00a0C<\/i>\u00a0will increase the number of training errors, while a large\u00a0C<\/i>\u00a0will lead to a behavior similar to that of a hard-margin SVM.” Joachims (2002), page 40<\/p>\n

5) Analyzing output in different cases<\/span><\/h3>\n

What if I vary dictionary size?<\/strong><\/p>\n

Varying dictionary size means changing the number of features (words). So, I wanted to explore the impact of having more features and what’s the limit of a good result based on confusion matrix<\/a>\u00a0result.<\/p>\n

I tested on size= {3000,5000,6000,7000} and discovered that at size = 7000, SVM classification starts slightly dropping (false identification) while Naive Bayes delivered same results despite the size variation.<\/p>\n

I think that at that point maybe target classes started overlapping or training data overfitting. I’m not yet sure about the explanation of the result.<\/p>\n

What if I try Gaussian and Bernoulli?<\/strong><\/p>\n

Obviously, introducing Bernoulli won’t help because as I explained above, it doesn’t provide enough information in our case, we need the count of words, not the presence\/absence of it.<\/p>\n

Multinomial NB: <\/span>
\n[[129 \u00a0\u00a01]<\/span>
\n [ \u00a09 121]]<\/span>
\nGaussian NB:<\/span>
\n[[129 \u00a0\u00a01]<\/span>
\n [ 11 119]]<\/span>
\nBernoulli NB:<\/span>
\n[[130 \u00a0\u00a00]<\/span>
\n [ 53 \u00a077]]<\/span><\/p>\n

As we can see, Multinomial NB outperformed both Gaussian NB and Bernoulli NB.<\/p>\n

What if I try GridSearch<\/a> on SVM in order to tune SVM parameters?<\/strong>
\nParams GridSearch: param_grid = {‘C’:[0.1,1,10,100,1000],’gamma’:[1,0.1,0.01,0.001,0.0001]}
\nBest params found: Gamma: 0.0001 ; C: 100<\/p>\n

Linear SVM:<\/span>
\n[[126 \u00a0\u00a04]<\/span>
\n [ \u00a05 125]]<\/span>
\nMultinomial NB:<\/span>
\n[[129 \u00a0\u00a01]<\/span>
\n [ \u00a09 121]]<\/span>
\nSVM:<\/span>
\n[[129 \u00a0\u00a01]<\/span>
\n [ 62 \u00a068]]<\/span>
\nGridSearch on SVM:<\/span>
\n[[126 \u00a0\u00a04]<\/span>
\n [ \u00a02 128]]<\/span><\/p>\n

SVM tuned parameters using GridSearch allowed to get better results.<\/p>\n

Conclusion<\/span><\/h4>\n

So, that was about my first steps in Email-Spam Filtering application! I hope that it’s helpful if you are thinking about starting a text-classification project! And I will continue sharing various reflections and experiments along the way. For next time, I will explore more the improvement\/change of training data and features.<\/p>\n

The project is on Github<\/a> too.<\/p>\n

Tell me about your experience with text-classification? In what did you apply it? What methods do you suggest to apply? What are the challenges?<\/em><\/strong><\/p>\n

Helpful Resources:<\/strong><\/p>\n

[1] Naive Bayes and Text Classification<\/a>.
\n[2]Naive Bayes by Example<\/a>.
\n[3] Andrew Ng explanation of Naive Bayes video 1<\/a>\u00a0and video 2<\/a>
\n[4] Please explain SVM like I am 5 years old<\/a>.
\n[5] Understanding Support Vector Machines from examples<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Since last few months, I’ve started working on online\u00a0Machine Learning Specialization provided by the University of Washington. The first course was about ML foundations, the second about Linear Regression, and the third course on which \u00a0I’m currently on track is about Classification. I liked the courses almost in every aspect as they\u00a0teach ML algorithms implementation […]<\/p>\n","protected":false},"author":1,"featured_media":1093,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sfsi_plus_gutenberg_text_before_share":"","sfsi_plus_gutenberg_show_text_before_share":"","sfsi_plus_gutenberg_icon_type":"","sfsi_plus_gutenberg_icon_alignemt":"","sfsi_plus_gutenburg_max_per_row":"","footnotes":""},"categories":[11,23,13],"tags":[18,38,20,37,36],"class_list":["post-1073","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-my-activities","category-projects","category-technology","tag-machine-learning","tag-naive-bayes","tag-python","tag-svm","tag-text-classification"],"_links":{"self":[{"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/posts\/1073","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/comments?post=1073"}],"version-history":[{"count":22,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/posts\/1073\/revisions"}],"predecessor-version":[{"id":1085,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/posts\/1073\/revisions\/1085"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/media\/1093"}],"wp:attachment":[{"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/media?parent=1073"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/categories?post=1073"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sarahmestiri.com\/index.php\/wp-json\/wp\/v2\/tags?post=1073"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}