Showing posts with label SVM. Show all posts
Showing posts with label SVM. Show all posts

Friday, April 23, 2021

Review - Machine Learning For Absolute Beginners - Oliver Theobold

Machine Learning For Absolute Beginners: A Plain English Introduction (Second Edition) 

Oliver Theobald

This book at Amazon

ISBN-10 : 1549617214

ISBN-13 : 978-1549617218

There is a wealth of information out there on the subject of Machine Learning (ML) especially over the last 10 years. There are books with catch-phrase titles but lacking in substance. Conversely there are books which dwell too deeply into certain kinds of programming framework for ML (nothing wrong with that). For a data scientist, there are not many books out there which cover a broad spectrum of ML, yet with substance on each topic. 'Machine Learning for Absolute Beginners' by Oliver Theobold seems to be one of these rare gems that covers ML broadly and capable of explaining important concepts without too much mathematics or code snippets. Despite the title, a seasoned data scientist may yet find this as a valuable reference.

Following a pleasant introduction to ML and an overview of a typical workflow, it discusses in detail the 3 classes of ML: Supervised, Unsupervised and Reinforcement Learning. It describes a special case of Reinforcement Learning known as Q-Learning. This is followed by a chapter on ML Toolbox where it discusses the data requirements, the infrastructure, the algorithms and the visualisation options that are available. Here it explores the various programming languages (Python, C, R, etc), the cloud platforms (Azure, AWS, Google) and the algorithm frameworks (Tensorflow, Caffe, Torch, etc).

The chapters on data preparation and feature selection covers a lot of the techniques as well as the motivation behind them. These include row and column compression, one hot encoding, binning, treatment of missing data, normalization and standardization. Partitioning of data and cross validation (k-fold) are discussed as the final stages of preparing the data before model training.

The next two chapters were devoted each to Regression and Clustering respectively. These are perhaps the two foundational families of algorithms which are also known outside the context of ML. The difference between Linear and Logistic regression conceptually and applicability to real scenarios were well illustrated. A very simple hand calculation shows how linear regression really works. Logistic regression is then explained by contrasting its usage with linear regression. Similarly the concepts and application for k-Nearest Neighbours and k-Means Clustering for supervised and unsupervised learning were well explained, including the reasonings behind the algorithms.

Machine Learning For Absolute Beginners: A Plain English Introduction (Second Edition) (Machine Learning From Scratch Book 1)

Bias and variance has a short chapter by itself and is enough to explain with sufficient details. Here is another example that brings this book above other ML books which just focus on coding. The understanding of the interplay between bias and variance, which is also emphasized in Andrew Ng's Machine Learning course, differentiates between a data scientist and an ML coder.

Next two chapters deal with the advanced algorithms Support Vector Machines (SVM) and Artificial Neural Networks (ANN). SVM is contrasted with Logistic regression, with explanation of the hyperplanes concept and the margins around the boundary for SVM. ANN concepts were explained from the basic components of neurons and activation functions, building up to the perceptron with mathematical examples for very simple cases. Multilayered Perceptron (MLP) and the various Deep Learning techniques were briefly discussed.

While ANN may be very effective in some cases, it is well known for its lack of explanability (except perhaps with recent developments in SHAP, ICE, LIME, etc). On the other hand Decision Trees has often been hailed for its high explainability nature. So a decent chapter is devoted to Decision Trees with simple mathematical calculations of entropy - which is key to decide on how to split the nodes of the tree. This is expanded to the advanced trees family including Boosting, Bagging and Random Forest and when to use them effectively. This is followed by a short chapter in Ensemble Modelling, where the Stacking method is also discussed.

The last three chapters, finally but rightly put, deals with introducing the development environment and the tool necessary for machine learning work. These include using the anaconda environent to program in the Python language and using the Panda library's dataframes. It also discusses the concept of model optimization and illustrated using the Grid Search technique. Towards the end there is a source code for a simple but complete Python code for an ML workflow, recommendation of other useful books, an Appendix on the introduction of Python but what I consider most useful is pointers to a few free datasets. 



Wednesday, July 03, 2013

Notes BigData

Notes BigData
===============

Definition
Web Intelligence and Big Data course
Why Big Data
Hadoop Ecosystem
MapReduce
Miscellaneous
Analysis


Definition
===========
Ref:  http://www.intel.com.au/content/www/au/en/big-data/unstructured-data-analytics-paper.html

- All history until 2003 - 5 exabytes
- 2003 to 2012 - 2.7 zettabytes
- data generated by more sources, devices, including video,
- data are UNSTRUCTURED, texts, dates, facts. Traditional Analytics are Structured Data (RDBMS).
- Analytics = Profit. Gartner survey - outperform competitors by 20% for those who use Big Data.
-



Web Intelligence and Big Data (WIBD) course
======================================
50 billion pages indexed by google.



More surprising events is better news.
- if event has prob p, then
    information = log_2 (p)  bits of information.

Mutual Information (MI) - between transmitted and received channels
- need to maximise MI
- eg mutual information between Ad$ and Sales
- eg adsense - given webpage, guess keywords.

IDF = inverse document frequency
- rare words make better keyword.
- IDF of Word = Log_2 (N / N_w)
  where N = total docs, N_w = number of word 'Word' in total docs.

TF = Term Frequency
- number of times the terms appear in that specific document.
- more frequent words (in that doc) make better keywords.
- TF = freq of w in doc d = n_w^d

TF-IDF = term freq x IDF = n_w^d x log_2 (N/N_w)
- words with high TF-IDF are good keywords.

Mutual Information between all pages and all words is prop to
   SUM_d  Sum_w  {  n_w^d x log_2 (N/N_w)  }

Mutual Information:  Input F -- Machine Learning -- Output B
Feature F, Behaviour B are independent.
Entropy H(F), H(B)
Mutual Information I(F,B) = SUM_f, SUM_b p(f,b) log { p(f,b) / p(f).p(b) }
Shannon: H(F) + H(B) - H(F,B)

WIBD - Naive Bayes
===================
Consider problem: P(BUY / r,f,g,c) where r,f,g,c are feature or keywords in web shopping.
Bayes Rule: P(B,R) = P(B/R).P(R) = P(R/B).P(B)
Naive Bayes assume r,f,g,c are INDEPENDENT
   - can derive Likelihood
      p(r/B)*p(c/B)*p(other features /B) ..... p(B)
      ---------------------------------------------  = L
      p(r/notB)*p(c/notB)*p(other features /notB) ..... p(notB)

      so if L > 1 we have a BUY, L < 1 then no Buy.



WIBD - Learn
==============
input X = x1,x2,...xn  (n-dimensional)
output y = y1,y2, ... ym
function f(X) = E[Y/X] expectation
              = y1*P(y1/X) + y2*P(y2/X)
Classification(video 5.2)
- eg X=size, head, noise, legs, Y={animal names}
- eg X= {like, lot}, {hate, waste}, {not enjoy}, Y = {positive, negative}
Clustering - unsupervised
- allow us to get classes from data. Need to choose right features.
- used when we DON'T know outputs relationship to start with.
- by Defn Clustering are regions MORE populated than random data
- add random data so that Po(X). So that r= P(X)/P0(X) is large means there is clustering
  then f(X)=E[Y/X]=r/(1+r) y=1 for real data, y=0 for added random uniform data.
- find things that do together to form a cluster. Eg Negative sentiment: hate, bad - but no one need to tell us they are Negative to start with.
- other means of clustering: k-means, LSH
Rules
- finding which features are related (correlated) to be each other. ie trying to cluster the features, instead of clustering the data.
- compare data which are independent features: Po(X) =  P(x1) * P(x2) * ... * P(xn)
  where x1 = chirping, x2 = 4 legged, etc xi={animal features}
  eg P(Chirping) = number of chirping / number of total Data
- Associative Rule Mining
  if there are features, A,B,C,D, want to infer some rule, eg A,B,C => (infer) D
  high support P(A,B,C,D) > s;   technique is to find P(A)>s, P(B)>s etc first
  high confidence P(D/A,B,C) > c
  high interestingness P(D/A,B,C) / P(D)  > i
- Recommendation of books - customers are features of books and vice versa.
  Use latent Models: matrix m x n = m x k TIMES k x n
                  eg people x books = people x genre TIMES genre x books
  NNMF - Non-negative

features: unemployment direction, interest rate direction, fraud

WIBD - Connect
===============
Logic Inference
if A then B    is SAME as ~A OR B  
Obama is president of USA:  isPresidentOf(Obama, USA) - predicates, variables
IF X is president of C  THEN X is leader of C:    IF  isPresidentOf(X,C) THEN isLeaderOf(X,C)
Query: If K then Q, consider that the query means ~K OR Q is TRUE,
       also same as K AND ~Q is FALSE.
       So proving K AND ~Q is FALSE, this means If K Then Q.

WIRD - Prediction
==============
Linear Least Squares Regression:
  x(i,j) with j features to predict, i-th data points with results yi for the i-th point
  Minimizing f(x) = E(y/X) is same as minimizing error = E(y-f(x))^2,  
  ... so let f(x) = xT.f   where f is the features vector of unknowns.
  Finding vector derivative and equate to zero ->  xT.x.f - xT.y = 0
  R^2 used to measure linear regression
Non - Linear correlation
  - Logisitic Regression - f(x) = 1 / (1 + exp(-f^T. x))
  - Support Vector Machines - Data may be high order correlated, eg parabolic correlation etc.
 Neural Networks
   - linear least squares
   - non-linear like logistic
   - feed-forward, multilayer
   - feed-back, like belief network
Which Prediction technique
FEATURES   TARGET  CORRELATION         TECHNIQUE
num        num     stable/linear       Linear Regression
cat        num                         Linear Regression, Neural Networks
num        num     unstable/nonlinear  Neural Networks
num        cat     stable/linear       Logistic Regression
num        cat     unstable/nonlinear  Support Vector Machines
cat        cat                         Support Vector Machines, Naive Bayes, Other Probabilistic Graphical Models



Why Big Data
==============
eg why Google(MapReduce), Yahoo(PIGS), Facebook(Hive) have to invent new stack
Challenges
1. Fault tolerance
2. Variety of Data Types, eg images, videos, music
3. Manage data volumes without archiving. Traditional need archives.
4. Parallelism was an add-on
Disadvantages
1. Could not scale
2. Not suited for compute-intensive deep analytics, eg in web-world
3. price-performance challenge. uses commodity hardware, open-source


Hadoop Ecosystem (See NotesHadoop)
===================================

MapReduce (See NotesHadoop)
=============


Miscellaneous
==============
About our speaker: Bio: Ross is Chief Data Scientist at Teradata and currently works with major clients throughout Australia and New Zealand to help them exploit the value of ‘big data’. He specialized in deployments involving non-relational, semi structured data and analyses such as path analysis, text analysis and social network analysis. Previously, Ross was deputy headmaster of John Colet School for 18 years before working as a SAS analyst, a business development manager at Minitab Statistical Software and founder and lead analyst at datamilk.com.

Ross Farrelly has a BSc (hons 1st class) in pure mathematics from Auckland University, a Masters in Applied Statistics from Macquarie University and a Masters of Applied Ethics from the Australian Catholic University.

Analysis
=========
path analysis
text analysis
social network analysis
natural language processing