xTechNotes - Technical Computing Programming Notes

Thursday, May 17, 2012

Notes R

NotesR
=======
This is for Windows mainly. However, many of the notes may apply to Linux as well.

Contents
=========
Setup / Install
R Upgrade new version
R - changs in 2.15.0
Running R
Quick Commands
R.matlab
R object types
Arrays
Packages - Manual Install
Packages
R - Stat functions
R - Time Series functions
Exploratory Data Analysis
Plotting on multiple windows
Writing to HTML files
List
Sets Operations
R - functions
3D plots
Plotting features
ECDF ecdf Empirical CDF
data.frames
Reading Data Files
serialIndepTest
[R] Operators
RUnit - Testing
Linear Least Squares Regression
R Interoperability
Calling Fortran from R
Fitting Distributions using VGAM::vglm and others
Error Handling
Sweave
Global variables and scope

Setup / Install
================
Mirror Site: http://cran.ms.unimelb.edu.au/
Installed in directory:
C:\Program Files\R\R-2.12.1\bin\i386

To install additional packages, use the GUI to download and install.

Help files:
http://127.0.0.1:21879/library/utils/html/help.html
C:\Program Files\R\R-2.12.1\doc\html\index.html

Appendix A in manual is a helpful starter.

Path - typical installation path is: C:\Program Files\R\R-2.12.1\bin
Include the path in the PATH variable or another envrionment variable

R Upgrade new version
======================
The R installation is setup such that multiple versions can exist on Windows.
For example it is possible to have installations at:
C:\program files\R\R-2.12.1
C:\program files\R\R-2.12.2
To perform the upgrade, suppose R-2.12.1 already exist and you wanted the new version R-2.12.2. The steps are:
1. Download R-2.12.2-win.exe
2. Close any R applications that is running.
3. Install by running R-2.12.2-win.exe
4. In Windows Explorer, go to C:\program files\R\R-2.12.1\library.
Highlight all folders and COPY (Ctrl-C)
5. Go to the new installation path C:\program files\R\R-2.12.2\library.
Paste (Ctrl-V) BUT DO NOT OVERRIDE existing folders.
This action will copy the libraries YOU installed in the old version to the new version BUT does not delete the base libraries that exist already in the new installation.
6. Run R-2.12.2, go to Packages -> Update packages...
7. If you wish, you can try uninstalling the older version. Please note that I have not done this so I cannot say if this step will affect the new version or not.

R - changes in 2.15.0
======================
Some changes to R functions which may are incompatible with previous versions are:
old: fisk(link.a = "loge", link.scale = "loge", earg.a=list(), earg.scale=list(), init.a = NULL, init.scale = NULL, zero = NULL)
new: fisk(lshape1.a = "loge", lscale = "loge", eshape1.a = list(), escale = list(), ishape1.a = NULL, iscale = NULL, zero = NULL)

old: VGAM::psinmad( a=..., q.arg=...) changed to
new: VGAM::psinmad( shape1.a=..., shape3.q=...)

old: VGAM::qfisk( a=....)
new: VGAM::qfisk( shape1.a=...)

Ref: http://www.r-project.org/
C-LEVEL FACILITIES:

o Passing R objects other than atomic vectors, functions, lists and
environments to .C() is now deprecated and will give a warning.
Most cases (especially NULL) are actually coding errors. NULL
will be disallowed in future.

.C() now passes a pairlist as a SEXP to the compiled code. This
is as was documented, but pairlists were in reality handled
differently as a legacy from the early days of R.

o call_R and call_S are deprecated. They still exist in the
headers and as entry points, but are no longer documented and
should not be used for new code.

Running R
==========
Two ways to use the R software
1. Rgui.exe - this is the windowing environment.

2. Rterm.exe - this is a command terminal environment.
- run by typing in any cmd terminal:
C:\Program Files\R\R-2.12.1\bin\i386

3. Running in batch mode:
Rterm.exe --no-restore --no-save < infile > outfile 2>&1

Alternatively, creaate a file called R.bat with the following content:
"c:\Program Files\r\R-2.13.1\bin\x64\Rterm.exe" --vanilla
To run,
R < test.R > test.out 2>&1
Or,

start cmd /K "R.bat < aaa.R > aaa.out 2>&1" - new CMD window remains open
start cmd /C "R.bat < aaa.R > aaa.out 2>&1" - new CMD window closes

4. Running scripts from Windows CMD terminal.
- set windows environment variable to installation path, eg:
Rbin=C:\Program Files\R\R-2.12.1\bin
- go to the dir with R file eg file called Initialize.R
- Run: "%Rbin%\Rscript" Initialize.R
- OR: START "runMe" cmd /K "%rbin%\Rscript" runMe.R

5. Running scripts from inside an R console.
- source("Initialize.R", print.eval = TRUE)
the print.eval = TRUE ensures that evaluation in script file is printed to screen

6. Running Parallel from Windows CMD terminal
- START "someName" "%rbin%\Rscript" someScript1.R
- START "someName" "%rbin%\Rscript" someScript2.R
the START is a windows command that forks open a separtate Windows terminal

Quick Commands
===============
license(), licence() - for distribution details.
contributors() - list of contributors
citation() - on how to cite R or R packages in publications.
demo() - for some demos
help() - for on-line help
help.start() - for an HTML browser interface to help.
help(par), ?par - help for function par
??par - multiple lists of help for par
q() - to quit R.
<command> - typing just the command line reveals the code
<command()> - executes the function.
ls() - list variables in the workspace
objects() - list of objects in the workspace
rm(a, b) - removes object a and b in the workspace
rm(list=ls()) - WARNING: removes all variables
rm(a, envir = globalenv()) - removes object from the Global Environment
search() - list the loaded packages
getAnywhere(x) - find which packages x belong too, good when multiple packages have same function name.
# - comments symbol
list.files() - list the files in the current directory
getwd() - get the path of the current directory\
setwd("c:\\data") - sets the working directory to c:\data
print(<r commands>) - prints output to screen, useful for R scripts.
paste("aa",xx,sep="") - concatenate strings; note xx may be a number
match(), pmatch(), grep - string matching
y<-unlist(strsplit(x, "_")) - parse or split the string x, using delimiter as "_". Access each item by y[0], y[1], etc.

which(data==const) - mask. Returns the indices where condition is true.
which(data==const, arr.ind=TRUE) Returns indices properly according to dimensions. If arr.ind=FALSE (default),
then even for multi dim array, it returns the index cumulated in ONE dimension only.
which.max - get the index of the maximum value of a vector.
max.col - get the index of the maximum value of a matrix (have not tried)
options(digits=xxx) - change precision display, where xxx is the number of digits required.
format(aaa, digits=9) - change precision display
sprintf("%.14f",aaa) - change precision display
calibrate::textxy - special text formatting (also see text in core package)
source("filename.R") - Batch: runs commands in file filename.R
sink("filename.R") - Batch: outputs to file filename.R, MUST USE PRINT COMMAND to print to file using SINK.
sink() - return output to terminal
write(t(x), file="a.txt", ncolumns=31, append=TRUE, sep="\t")
- write 2D matrix x (need transpose t), to file called a.txt, display in 31 columns, with tab separation and append to existing file.
- write to console by putting argument file="".
save(x, file=' ') - to save a variable to file, in binary format. See sink() command to save file in text format.
load(file=' ') - to load data, in binary format. Note: Variables are automatically loaded and overwrite existing variables.
file.exists('filepath') - checks if a file exists and returns TRUE/FALSE.
memory.size() - set/get the memory size in MB
gc() - garbage collection
a %in% b - test to see if element a is in vector b.
compare(a,b) - Package: compare. Compares objects a and b to see if they are the same.
runif(n, min=0, max=1) - Random Uniform Distribution - draw n points randomly.
set.seed(x) - Sets the seed for random generators.
table(xWords) - split data into frequency bins
c(rbind(vecA, vecB)) - interleave 2 vectors of data
file.exists('filename') - checks if the file exists
Sys.Date() - returns system date in format 'yyyy-mm-dd'
Sys.time() - returns date and time in format 'yyyy-mm-dd hh:mm:ss: EST'
system.time([blah]) - time it takes to execute blah.
process.time() - call this, run R statement, then call again and take difference between two times.
gc.time() - garbage collection time.

str(xxx) - information about the components of xxx
summary(xxx) - information summary about xxx
names(xxx) - information about the names of the components of xxx.

t(x) - Transpose of matrix x.
m1 <- matrix(1:12, nc=3) - create a matrix with 3 columns using the 12 data points
Data are distributed columnwise by default.
m1 <- m1[-4,] - remove the 4th row from a matrix.
m2 <- rbind(m1, c(1,2,3)) - add items 1,2,3 as a new row of matrix m1.
1:5 - like linspace, generates {1,2,3,4,5}
seq(a,b, by=c) - like linspace, generates sequence from a to b with step c.
rep(x, [times|each]) - replicates vector x with various options.
a %% b - the remainder of the number of times a is divisible by b
a %/% b - the modulus or the number of times a is divisible by b

dyn.load("a.dll", local=FALSE) - load the dll into R. The local=FALSE puts it in Global space rather than local space.
dyn.unload() - unload the dll from R, required if the dll needs to be recompiled.
is.loaded() - check whether a subroutine name has been loaded into R.
R.version - to list the system information R is run on.

sapply(X, FUN, ...) - a user friendly version of lapply. It applies a function to the list of objects.
- if X is a list of collection, eg x <- list(a = 1:10, beta = exp(-3:3), logic = c(TRUE,FALSE,FALSE,TRUE)) - then sapply(x, mean) will return 3 means, one for each group.
- if X is a vector of elements, then the function is applied to each element.
http://www.ats.ucla.edu/stat/r/library/advanced_function_r.htm
as.vector - vectors have NO dimension. Can us this to make num[1:14(1d)] into num[1:14]

months(aDate) - return Month in characters, abbreviate=TRUE to get short version.
as.Date(aDate, format) - translate in R Date object for input with specified format.
for(ii in 1:10){} - for loop control structure
for(ii in c(1:10){} - for loop control structure - using a numbers from a list
next - to skip to next iteration in for loop.
break - to break out of a for loop.

is.null(arg) - tests if the argument is NULL
is.nan(arg) - tests if the argument is NaN
is.finite(arg) - tests if the argument is finite
is.infinite(arg) - tests if the argument is infinite
.Machine$double.eps - gives machine EPSILON value
exists('aVar') - tests if the variable called aVar is defined or not.

knots(ecdf(X)) - retrieves distinct values of ecdf
setdiff(A,B) - data in A, not in B

R.matlab
=========
it appears the R package is not compatible with all versions of .mat file
you must save file as v6: save('D:\DATA\QARiskApp\mloss-v6.mat','mLoss','mLogLoss','-v6')

readMat(file.path("<mat file>")) reads the contents of "mat file"
bs <- readMat(file.path("<mat file>")) reads the contents of "mat file" and store in a structure "bs"

If the data stored in BS are made of many different types of data, eg integer, vectors, and come in different shapes, eg scalar or vectors, then the relevant commands are:
ll - info about the general structures available
bs[[1]] - first item of the structure. This can be scalar, 1D, 2D, etc matrices.
bs$<var1> - same as above, accesses the first component of the structure.

R object types
===============
Ref: http://cran.r-project.org/doc/contrib/R_language.pdf
http://cran.r-project.org/doc/manuals/R-lang.html#Basic-types

The types of objects are:
vectors, arrays, factors, lists, data frames, functions

Intrinsic attributes of objectts are:
mode(object)
length(object)

To change the type of an object, use:
as.characters(object)
as.integers(object)
as.<etc..>(object)

Objects can change size dynamically, or
by using length: length(object) <-3 # sets the array to new size 3
by resizing itself: object <- object[2* 1:5]

Objects have the following attributes: class, comment, dim, dimnames, names, row.names and tsp
attributes(object) lists all the attributes
attr(object, "dim") get or set the attribute "dim"

Arrays
=======
Any vector can be made into array if its "dim" attribute is defined. This gives the object a multi-dimensional shape.

The various ways of defining arrays are:
Z <- array(h, dim=c(3,4,2))
Z <- h ; dim(Z) <- c(3,4,2)
scal = "blah" ; zz <- array(scal, dim=c(3)); # resize into zz = { blah, blah, blah }
Z <- array(0, dim=0) # to create an array of zero length, it is of type numeric(0)

Outer product can be defined as:
ab <- a %o% b
ab <- outer(a, b, "*")
f <- function(x, y) cos(y)/(1 + x^2); z <- outer(x, y, f)

tapply() function - provides masking functionality.
Example:
state <- c("tas", "sa", "qld", "nsw", "nsw", "nt", "wa", "wa",
"qld", "vic", "nsw", "vic", "qld", "qld", "sa", "tas",
"sa", "nt", "wa", "vic", "qld", "nsw", "nsw", "wa",
"sa", "act", "nsw", "vic", "vic", "act")
statef <- factor(state)
> statef
[1] tas sa qld nsw nsw nt wa wa qld vic nsw vic qld qld sa
[16] tas sa nt wa vic qld nsw nsw wa sa act nsw vic vic act
Levels: act nsw nt qld sa tas vic wa
levels(statef)
[1] "act" "nsw" "nt" "qld" "sa" "tas" "vic" "wa"
incomes <- c(60, 49, 40, 61, 64, 60, 59, 54, 62, 69, 70, 42, 56,
61, 61, 61, 58, 51, 48, 65, 49, 49, 41, 48, 52, 46,
59, 46, 58, 43)
incmeans <- tapply(incomes, statef, mean)
giving a means vector with the components labelled by the levels (using function mean())
act nsw nt qld sa tas vic wa
44.500 57.333 55.500 53.600 55.000 60.500 56.000 52.250
stderr <- function(x) sqrt(var(x)/length(x)) # user defined function
incster <- tapply(incomes, statef, stderr)
> incster
act nsw nt qld sa tas vic wa
1.5 4.3102 4.5 4.1061 2.7386 0.5 5.244 2.6575

Subarray - let A = array of size [3,3]
A[,2] = second column of A
A[2,] = second row of A

Packages - Manual Install
==========================
1. Download packages from: http://cran.ms.unimelb.edu.au/
2. In R, select Packages -> Install Packages from Local Zip file
3. Choose the zip file from the location where it was downloaded.

Packages
=========
sessionInfo() - tells the packages loaded.
library() to see which packages are installed
library(boot) to load the package called boot - gives error if package not found
require(boot) to load the package called boot - gives warning if package not found
library(help = evir) to see the list of functions in the package
search() to see which package are loaded
loadedNamespaces() to see the packages loaded but not on the search list.
<lib>::<func>() to specify overloaded function using its library name to avoid confusion, eg base::Version()

QQ Plots {stats} : qqnorm, qqline, qqplot
importFrom(foo, f, g) loads only the functions f,g from packages foo
import(foo, bar) loads all functions which are exported from the packages foo and bar.
detach("package:R.matlab", unload=TRUE) remove from search() path a data.frame that has been attached or a package that was attached by library.
unloadNamespace("ns") unloads the namespace ns.

unloadNamespace("R.oo")
R.oo::Class()

detach("package:R.methodsS3", unload=TRUE)
removes R.methodsS3 from search() and loadedNamespaces()

Calling a function with package notation, eg copula::genFun
will bring package copula back into loadedNamespaces() but not in search().
unloadNamespace("copula") - will remove it again from loadedNamespaces()

library(copula) - adds copula to search() and loadedNamespaces()
unloadNamespace("copula") - removes it from search() AND loadedNamespaces()
genFun() - calling has no effect
copula::genFun - brings copula into loadedNamespaces()

library(copula) - adds copula to search() and loadedNamespaces()
detach("package:copula") - removes it from search() but NOT REMOVEd from loadedNamespaces()
genFun() - calling has no effect
copula::genFun - is OK

To use packages with Namespaces
- no need to use library() or any other way to load functions
- in the code, simply call [package]::[function] # note the double colon
- when finished, call: unloadNamespace("[package]")
- using this method loads the package in and out of the Namespaces. It has not been added in the search path.

To use packages with NO Namespaces
- call the library by: library([package]). This will load the package into the search path and add to the Namespaces. But since the package has no namespace, nothing is added to the Namespaces.
- in the code, just call the function using its name: [function](arguments)
- when finished, call : detach("[package]:[function]") # note the single colon. This will remove it from the search path. Since it has no Namespace, there is no point in calling: unloadNamespace().

Searching rules with Namespaces
It searches in package namespace first, then among the functions imported by that package, then R base, and lastly in the “normal” search path (as from ‘search()’). See “Writing R Extensions”, last paragraph before the end of sub-section. Consequently, the correct version of ‘set.vertex.attribute’ is used.

Namespace REF: http://www.r-bloggers.com/namespaces-and-name-conflicts/

R - Stat functions
===================
runif - Uniform random generator
rt - Student-T random generator
rexp - Exponential random generator
rweibull - Weibull random generator
rgamma - Gamma random generator
rpareto - Pareto random generator http://www.commanster.eu/rcode.html, http://hosho.ees.hokudai.ac.jp/~kubo/Rdoc/library/rmutil/html/Pareto.html

R - Time Series functions
==========================
This section is taken from http://cran.r-project.org/doc/contrib/Ricci-refcard-ts.pdf
written by Vito Ricci

INPUT
cycle(): gives the positions in the cycle of each observation (stats)
deltat(): returns the time interval between observations (stats)
end(): extracts and encodes the times the last observation were taken (stats)
frequency(): returns the number of samples per unit time (stats)
read.ts(): reads a time series file (tseries)
start(): extracts and encodes the times the first observation were taken (stats)
time(): creates the vector of times at which a time series was sampled (stats)
ts(): creates time-series objects (stats)
window(): is a generic function which extracts the subset of the object 'x' observed between the times 'start' and 'end'. If a frequency is specified, the series is then re-sampled at the new frequency (stats)

TS DECOMPOSITION
decompose(): decomposes a time series into seasonal, trend and irregular components using moving averages. Deals with additive or multiplicative seasonal component (stats)
filter(): linear filtering on a time series (stats)
HoltWinters(): computes Holt-Winters Filtering of a given time series (stats)
sfilter(): removes seasonal fluctuation using a simple moving average (ast)
spectrum(): estimates the spectral density of a time series (stats)
stl(): decomposes a time series into seasonal, trend and irregular components using 'loess' (stats)
tsr(): decomposes a time series into trend, seasonal and irregular. Deals with additive and multiplicative components (ast)

TESTS
adf.test(): computes the Augmented Dickey-Fuller test for the null that 'x' has a unit root (tseries)
Box.test(): computes the Box-Pierce or Ljung-Box test statistic for examining the null hypothesis of independence in a given time series (stats)
bds.test(): computes and prints the BDS test statistic for the null that 'x' is a series of i.i.d. random variables (tseries)
bptest(): performs the Breusch-Pagan test for heteroskedasticity of residuals (lmtest)
dwtest(): performs the Durbin-Watson test for autocorrelation of residuals (lmtest)
jarque.bera.test(): Jarque-Bera test for normality (tseries)
kpss.test(): computes KPSS test for stationarity (tseries)
shapiro.test(): Shapiro-Wilk Normality Test (stats)

STOCHASTIC MODELS
ar(): fits an autoregressive time series model to the data, by default selecting the complexity by AIC (stats)
arima(): fits an ARIMA model to a univariate time series (stats)
arima.sim(): simulate from an ARIMA model (stats)
arma(): fits an ARMA model to a univariate time series by conditional least squares (tseries)
garch(): fits a Generalized Autoregressive Conditional Heteroscedastic GARCH(p, q) time series model to the data by computing the maximum-likelihood estimates of the conditionally normal model (tseries)

GRAPHICS
lag.plot: plots time series against lagged versions of themselves. Helps visualizing "auto-dependence" even
when auto-correlations vanish (stats)
monthplot(): plots a seasonal (or other) subseries of a time series (stats)
plot.ts(): plotting time-series objects (stats)
seaplot(): plotting seasonal sub-series or profile (ast)R functions for time series analysis by Vito Ricci (vito_ricci@yahoo.com) R.0.5 26/11/04
seqplot.ts(): plots a two time series on the same plot frame (tseries)
tsdiag(): a generic function to plot time-series diagnostics (stats)
ts.plot(): plots several time series on a common plot. Unlike 'plot.ts' the series can have a different time bases, but they should have the same frequency (stats)

MISCELLANEOUS
acf(), pacf(), ccf(): the function 'acf' computes (and by default plots) estimates of the autocovariance or autocorrelation function. Function 'pacf' is the function used for the partial autocorrelations. Function 'ccf' computes the cross-correlation or cross-covariance of two univariate series (stats)
diff.ts(): returns suitably lagged and iterated differences (stats)
lag(): computes a lagged version of a time series, shifting the time base back by a given number of observations (stats)

Exploratory Data Analysis
==========================
summary(data) Min, 1stQ, Median, Mean, 3rd Q, Max
fivenum(data) Returns Tukey's five number summary (minimum, lower-hinge, median, upper-hinge, maximum) for the input data.
stem(data) Stem and leaf plot

Plotting on multiple windows
=============================
# Create 3 plots
dev.new() # Or X11()
dev.1 <- as.integer(dev.cur())
dev.new()
dev.2 <- as.integer(dev.cur())
dev.new()
dev.3 <- as.integer(dev.cur())

x <- seq(1, 100, 0.1)

# Switch to device 1
dev.set(dev.1)
plot(x, sin(x), "l")
# Switch to device 3
dev.set(dev.3)
plot(x, cos(x), "l")
# Add something to graph #1
dev.set(dev.1)
points(x, cos(x), "l", col="red")

Writing to HTML files
=======================
This functionality of writing to HTML files are facilitated by either of the following packages:
i) HTMLutils - which depends on R2HTML
ii) hwriter

The functionalities are best described using examples.
The examples here are based on the hwriter package only.

<code>
pHand <- openPage(fileOut, title=title)
hwrite(title, page=pHand, center=TRUE, heading=1)
hwrite('',pHand, br=TRUE)

hwriteImage(fileOut_hist, pHand, br=TRUE)
hwrite('Histogram',pHand, br=TRUE)

closePage(pHand, splash=TRUE)
</code>

The html file is created using the openPage() function and finishes off using closePage() function.
pHand - The file handle. This is needed by closePage as well as other html functions to identify which file to write to.
hwrite() - the function used to write html text. Note there are many arguments in hwrite() to control HTML parameters like: p, br, heading, center, etc.
hwriteImage() - the function to write an image to the html file. In this example, the path of the image is stored in the string fileOut_hist.

List
=====
A list in R can be made up of different data types (numbers, strings) as well as different size objects.
An example is :
Lst <- list(name="Fred", wife="Mary", no.children=3, child.ages=c(4,7,9))
In the example above, the list has got 4 components which are: name, wife, no.children, child.ages
The components can be accessed as Lst$wife or Lst[[2]]
The fourth component is a vector and each elements {4,7,9} can be accessed as:
Lst[[4]][1], Lst[[4]][2], Lst[[4]][3] OR
Lst$child.ages[1], Lst$child.ages[2], Lst$child.ages[3] OR
Lst[["child.ages"]][1], Lst[["child.ages"]][2], Lst[["child.ages"]][3] OR
Creating list: list(name1=obj1, name2=obj2, ....)
Removing elements from a list by index : lll[-c(2,3)], remove second and third elements from the list.
Removing elements from a list by value : lll[lll != 'element']
Extracting elements from a list: lll[c(2,3)] extracting the second and third elements from the list.
Append an item (aaa) to an existing list(lll): append(lll, list(aaa))
Append an item to an 1D array of list
aa <- array(list(), dim=2) # create an array of list items, initially all items are NULL
aa[1] <- c(3.2) # so aa[1,2][[1]] = 3.2
aa[1] <- list(c(aa[1][[1]], 3.4)) # dereference the list, then concatenate, then make into a list again.
Append an item to an array of list
aa <- array(list(), dim=c(2,2)) # create an array of list items, initially all items are NULL
aa[1,2] <- c(3.2) # so aa[1,2][[1]] = 3.2
aa[1,2] <- list(c(aa[1,2][[1]], 3.4)) # dereference the list, then concatenate, then make into a list again.

To transform a list (ll) into just values, with no names:
ull <- unlist(ll)
names(ull) <- NULL

Dynamic reduction of list: This scans through a list, and at the same time removing specific elements of the list.
<code>
while(length(qList) != 0) {
... blah ...
# Update the qList by removing the certain elements, or rather choosing only specific elements to remain.
qList <- qList[qList != as.character(dataSet$xposId)]
... blah ...
} # end while
</code>

Sets Operations
================
is.element(elem, set) - test if element is in the set
union(x, y)
intersect(x, y)
setdiff(x, y)
setequal(x, y)
a & b - element wise comparison, say a is {TRUE, FALSE, TRUE}, b is {FALSE, FALSE, TRUE}, then answer is {FALSE, FALSE, TRUE}

R - functions
===============
The example below best illustrate the structure of a function in R. Here are the things to note.
- the name of the function in the example is myFunction
- the argument list (a,b) are INPUT arguments. They are placed next to the 'function' keyword.
- the output of the function is the most tricky part in R. Essentially, the output must be the last variable assignment in the function. So the last line has the variable 'out'.
- to produce multiple outputs, one way (as illustrated), is to create a list called 'out'. Then at the very last line, aggregate all your desired output variables into the list.

<code>
myFunction <- function(a, b){
out <- NULL # initialize output

c <- a + b
d <- a - b

# Final Result OUTPUT
out <- list(mySum=c, myDiff=d)
}
</code>

When calling the function, assign the function to a variable or handler,
result <- myFunction(2.3, 5.1)

To access the multiple output variables:
result$mySum
result$myDiff

3D plots
=========
The 3D plotting functions are:
image(x,y,z, ...) - heat plot
contour(x,y,z, ...) - contour plot
persp(x,y,z, ...) - wireframe plot

x = {1:rows}, y = {1:cols} are 1D vectors
z[1:rows, 1:cols] is the 2D matrix

3D Cloud plot example:
Data type must be
#str(data )
#'data.frame': 14880 obs. of 4 variables:
# $ x : num [1:14880(1d)] 1 2 3 4 5 6 7 8 9 10 ...
# $ y : num [1:14880(1d)] 1 1 1 1 1 1 1 1 1 1 ...
# $ z : num [1:14880(1d)] 1 1 1 1 1 1 1 1 1 1 ...
# $ val: Factor w/ 3 levels "-1","0","1": 1 1 1 1 1 1 1 2 2 2 ...
The plot function call is like:
<code>
thisPoints = Rows(trellis.par.get("superpose.symbol"), 1:3)
thisPoints$pch <- c(1,3,2)
thisPoints$col <- c("#0080ff", "#ff00ff", "#ffff00")

cloud(cRG ~ cBU * cRT, data = data , groups = val, screen = list(z = 20, x = -70),
perspective = FALSE, zoom = 1.0, pch = thisPoints$pch , col = thisPoints$col,
key = list(title = "Cluster", x = .15, y=.85, corner = c(1,-0.3),
border = TRUE,
points = thisPoints,
text = list(levels(data$val))))
</code>
Parameters like 'screen' are found under ?panel.cloud
show.settings() - will reveal the settings for "superpose.symbol"
?points - to see more options for points parameter
pch - controls the symbols for the points eg 1=circle,2=triangle,3=plus
col - color in Hex code

Plotting features
====================
All the parameters here appear as "plot( .... <param> ....)" unless otherwise specified:

plot(... xlim=c(-1, 10) ...) - specify the limit / boundary on the x-axis
text(x,y,labels,...) - specify text labels in a plot
lines(xvec, yvec) - draw line between 2 points
abline - draw line across whole graph using m,c from y=mx+c
plot(.... type='l' ...) - type: l=lines, p=points
plot(.... las= .....) - orientation of axis labels
plot(.... main='title' ....) - Title for the graph

To control the margin around the plot, specify the margin specification first:
par(oma=c(4,2,2,2))
barplot(.....)

Plotting to PNG files &
Plotting Empirical CDFs
# let vData be a vector of data point
aECDF <- ecdf(vData) # generate a ECDF object called aECDF
summary(aECDF) # summary stats for the ECDF
png("filename.png", bg="transparent")
plot(aECDF, main="title of graph", ylab="ylabel", xlab="xlabel")
dev.off()
# more examples
x10<-seq(1,10)
F10 <- ecdf(x10)
plot(F10)
plot(F10(x10), xlim=c(0,12))
plot(F10, xlim=c(0,12))
plot(x10, F10(x10), xlim=c(0,12), verticals=TRUE)
# another example
xData <- rweibull(100, shape=0.5)+3.0
fdata <- ecdf(xData) # note fdata is a ecdf function
plot(fdata, xlim=c(0,25)) # special plot takes in the ecdf function
plot(xData, fdata(xData), xlim=c(0,25)) # equivalent to using plot on the ecdf function

Adding points to existing plots:
- use the points() function, eg.
points(vecX, vecY, ....)

Plotting 2 or more data in one graph
plot(x), par(new=TRUE), plot(y)

ECDF ecdf Empirical CDF
==========================
aa <- c(1,2,3,4,5)
bb <- c(1,1,2,2,3,3,4,4,5,5)
plot(ecdf(aa))
par(new=TRUE)
plot(ecdf(bb))
### This results in identical ecdf

cc <- c(1,2,2,3,3,4,5)
plot(ecdf(aa))
par(new=TRUE)
plot(ecdf(cc))
### These two plots are different. The gaps are 1.0/7 ~ 0.148
### For the ecdf(cc), the x values are 1,2,3,4,5, the y values are: 0.14, 0.42, 0.71, 0.86, 1.0

dd <- c(2,3,3,2,5,1,4)
plot(ecdf(cc))
par(new=TRUE)
plot(ecdf(dd))
# Sorting has no effect on ecdf(), the data would be sorted by ecdf() anyway.

stepfun()
eg. myfun <- stepfun(1:10, cumsum(c(0, rep(0.1, 10))))
- first argument is x values, second argument starts at zero and is one element longer
- 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 x values
- 0.0, .1, .2, .3, .4, .5, .6, .7, .8, .9, 1.0 y values
stepfun() plot looks identical to ecdf plot, but with step lines drawn.

data.frames
=============
1. Creating data.frames from existing mixed type array.
r1 <- c(2,1, 0.44)
r2 <- c(23,11, 0.424)
dff <- data.frame(rbind(r1, r2))
colnames(dff) <- c("a", "b", "c")
rownames(dff) <- NULL removes the row names in a data frame

where the last row changes the default column names.

2. Creating data.frames from two vectors
age <- c(.....)
height <- c(.....)
village <- data.frame(age=age, height=height)

3. Accessing data.frames
village[1,2]
village["age", "height"]

4. Utility functions
nrow(village)

5. Read in data.frame from input file - see section on "Reading Data Files"

6. To hide, remove the first column which is row numbering, need to use
print(dataframe, row.names=FALSE)

Reading Data Files
====================
1. Reading data from text files into data frame.
Example: given the text data is in the following format:
RG BU RT cMap
1 1 1 -1
2 1 1 -1
3 1 1 -1
1 2 1 -1
2 2 1 -1

- say the data above is in the file called data.txt
- to read the data:
df <- read.table('data.txt', header=TRUE)
- the df structure would automatically be a data.frame.

serialIndepTest
================
1. d <- serialIndepTestSim(n=100, lag.max=3)
2. test <- serialIndepTest(x,d)
3. test
4. dependogram(test,print=TRUE)

Above are the steps to perform the serial independence tests using the serialIndepTest.
Step 1 creates the d object which is the result of simulation, which does not require any input data from the sample to be tested.
Step 2 performs the actual independence test.
Step 3 prints out the results. The Fisher result is better "tends to give the best results and was found to frequently outperform the test based on In".
Step 4. Plots and prints out the results.

Note that in Step 4, it tests all the subsets of lag of the original time series. The subsets are the Mobius decomposition. The original series is independent iff all the subsets are also independent. If any of the vertical line (test statistic) is below the dot(critical value), then that subset cannot reject the null hypothesis of independence. Only when the line is above the dot, then we reject the independence hypothesis for that subset.

Note (n - lag.max) >= 2

"Modeling Multivariate Distributions with Continuous Margins Using the Copula R package", Ivan Kojadinovic, Jun Yan, Journal of Statistical Software, May 2010, Vol 34, Issue 9.

[R] Operators
==============
- Minus, can be unary or binary
+ Plus, can be unary or binary
! Unary not
~ Tilde, used for model formulae, can be either unary or binary
? Help
: Sequence, binary (in model formulae: interaction)
* Multiplication, binary
/ Division, binary
^ Exponentiation, binary
%x% Special binary operators, x can be replaced by any valid name
%% Modulus, binary
%/% Integer divide, binary
%*% Matrix product, binary
%o% Outer product, binary
%x% Kronecker product, binary
%in% Matching operator, binary (in model formulae: nesting)
< Less than, binary
> Greater than, binary
== Equal to, binary
>= Greater than or equal to, binary
<= Less than or equal to, binary
& And, binary, vectorized
&& And, binary, not vectorized
| Or, binary, vectorized
|| Or, binary, not vectorized
<- Left assignment, binary
<<- Assigning to Global / R-Environment variables , see assignOps{base}
-> Right assignment, binary
$ List subset, binary

RUnit - Testing
================
RUnit is a unit testing package that allows each function to be tested automatically.
The following are instructions of how to set it up.

Ref: http://cran.r-project.org/web/packages/RUnit/vignettes/RUnit.pdf

1. Assume there is a function called c2f which needs to be tested. The c2f function can be like:
c2f <- function(c) return (9.5 * c + 32)

2. Create a file called "runitc2f.r" and write the test function as follows:
<code>
test.c2f <- function() {
checkEquals(c2f(0), 32)
checkEquals(c2f(10), 50)
checkException(c2f("xx"))
}
</code>

3. Create another file, with any name, as long as it is loaded using "source" command in R.
Then define the test suite as:

<code>
testsuite.c2f <- defineTestSuite("c2f",
dirs = file.path( <path to your test files> ),
testFileRegexp = "^runit.+\\.r",
testFuncRegexp = "^test.+",
rngKind = "Marsaglia-Multicarry",
rngNormalKind = "Kinderman-Ramage")
</code>

The name of this test suite file and the actual name of the defined testsuite does not matter much. But the most important names are the names of the test file which must be "runit..." and the actual test function names which must be "test....". These restrictions are defined in the definition of the test suite above.

4. If the test suite is defined in the file "testsuite.R", then to run the unit tests:
<code>
source('testsuite.R')
testResult <- runTestSuite(testsuite.c2f)
printTextProtocol(testResult)
</code>

5. To test Exception cases.
- in the actual function, there need to be a catch for the error and a call to the "stop" function
- in the runitXXXX.r file, the checkException is called with the function call as its argument, eg:
checkException( funcName(var1, var2, ...) )

Linear Least Squares Regression
================================
There is a few steps to perform the linear regression. The following example illustrates the various functions to facilitate regression analysis.

Example: Assume variable year, rate
cor(year, rate) - a first check for correlation

suppose we wish to fit the linear model as: rate = slope x year + intercept
fit <- lm(rate ~ year)
fit returns the coefficients for the linear equation such that
fit$coefficients[1] -> intercept
fit$coefficients[2] -> slope

To calculate the residuals:
res <- rate - slope x year + intercept
alternatively,
residuals(fit)

To plot the data and the regression line:
plot(year, rate)
abline(fit)

Finally, a whole lot of information can be obtained by:
summary(fit)

R Interoperability
===================
http://cran.r-project.org/doc/manuals/R-exts.html#Package-structure
http://www.revolutionanalytics.com/why-revolution-r/customer-testimonials.php
http://www.omegahat.org/RwxWidgets/ExampleDocuments/RC++Methods.pdf

using .Call - a souped up version of .C. It provides better ability to use R functions from within C code, and the C code can then be wrapped up to be used in R.
http://www.biostat.jhsph.edu/~bcaffo/statcomp/files/dotCall.pdf

Calling Fortran from R
=========================
http://www.stat.sc.edu/~grego/courses/stat740/handout.pcfortran.pdf
http://www.stat.umn.edu/~charlie/rc/
http://math.acadiau.ca/ACMMaC/howtos/Fortran_R.html

The steps to call Fortran subroutines from R is listed here using simple examples to illustrate the process.
A few notes before beginning:
- R can call Fortran Subroutines easily. Apparently it cannot call Fortran Functions directly. If a Fortran Function is needed, then it should be changed into a subroutine or create a wrapper function (this is a purely Fortran exercise and will not be detailed here).
- To call the Fortran from R, this example uses the .C() R function. Apparently the .Fortran() in R has some instabilities, so it will not be detailed in this example.

1. Create the Fortran subroutine, compile into a dll and expose the subroutines to be exported.
There are many ways and conventions of doing this. However, this example will only show the one that has been tested here and works. This example is based on Intel Fortran 11 on Windows. Also the naming convention to export the subroutine name is STDCALL (there are a few others, but this works here). The sample code is:

<code>
module modFtest
implicit none
contains

subroutine oneIntegerInput_dll(aaa)
!DEC$ ATTRIBUTES DLLEXPORT, STDCALL, ALIAS:'_oneIntegerInput_dll' :: oneIntegerInput_dll
!DEC$ ATTRIBUTES REFERENCE :: aaa
implicit none
integer(4), INTENT(IN) :: aaa
integer(4) :: bbb
bbb=aaa+1
end subroutine oneIntegerInput_dll

end module modFtest
</code>

A few things to be noted:
- The subroutine oneIntegerInput_dll is in a module called modFtest.
- The modFtest module is found in the CallFromR project. When compiled, the output would be a callfromr.dll.
- There can be multiple modules with multiple subroutines, constructed under this CallFromR project which would be accessible from its callfromr.dll
- The Intel Fortran (for Windows) way of exporting symbols is to use the DLLEXPORT attribute and the REFERENCE attribute to declare its variables.
- To be safe, just keep the Fortran variables to primitive data types. And declare all variables as REFERENCE.
- The accessibility of the Fortran variables can still be controlled via the INTENT statements.

2. Loading the DLL from R.
Within R, the dll can be loaded as:
dyn.load("callfromr.dll")

To unload, use:
dyn.unload("callfromr.dll")

If the dll is not in the current directory, the path may need to be provided. Ensure the path separation / or \ is correct for your system.
WARNING: If the dll is still loaded by R, then any attempt to recompile the Fortran code may fail because the dll is being used by R. The dll has to be unloaded before the dll can be recompiled by Fortran.

3. To use the subroutines from the dll, call as follows:
callfromr=.C("oneIntegerInput_dll", as.integer(3))

Note that the subroutine name is case sensitive. Although Fortran itself is not case-sensitive, the exporting of the name using DLLEXPORT has specified the name to be oneIntegerInput_dll with case-sensitivity.
The arguments of the subroutine need to be transformed from R, using as.integer() or something else.

4. Troubleshooting.
To check is a subroutine name has been loaded via the dll, type:
is.loaded("oneIntegerInput_dll")

5. Advanced arguments passing.
The above steps is the first steps to show Fortran - R compatibility. The section here will explore a combination of arguments that can be passed between Fortran and R with expanded examples.
WARNING:
a) In Fortran, DO NOT USE real(4). Instead use real(8)
b) For integers at least, when calling for R, need to cast the integer using as.integer(iii) in the .C(...) argument list.
c) Fortran array arguments must be declared like FCN(N), not FCN(1:*)

A. One integer in, one integer out
integer(4), INTENT(IN) :: aaa
integer(4), INTENT(OUT) :: bbb
The call can be made by:
callfromr=.C("twoIntInOut_dll", as.integer(3), as.integer(0))
The result is that callfromr will be a list of two integers.

B. Double Precision, one in, one out
REAL(8), INTENT(IN) :: aaa
REAL(8), INTENT(OUT) :: bbb
The call can be made by:
callfromr=.C("twoDblInOut_dll", as.double(3.1), as.double(0.0))

C. Vector of double precisions
integer(4), INTENT(IN) :: nn
REAL(8), INTENT(IN) :: aaa(nn)
REAL(8), INTENT(OUT) :: bbb(nn)
Setting up an R example
aaa <- seq(1,5)*1.2 # produces { 1.2 2.4 3.6 4.8 6.0 }
bbb<-seq.int(0, 0,length.out=5) # produces { 0,0,0,0,0 }
The call from R is:
callfromr=.C("vecInOut_dll", as.integer(length(aaa)), as.double(aaa), as.double(bbb))
The result will have 3 items, 1 integer, 1 input vector and 1 output vector.

#### Testing MleFitTruncLN
!DEC$ ATTRIBUTES REFERENCE :: K, L, X, w, parVal, parFlag, stderr, factor , cdf, sList

integer K !input, number of elements in the sample
double precision L !input, the known truncation level
double precision X(1:*) !input, sample data array
double precision w(1:*) !input array of weighting factors

double precision :: parVal(1:*) !output parameter values (mu and sigma)
integer parFlag(1:*) !input, option flag

double precision stderr(1:*) !output stderr(mu) and stderr(sigma)
double precision factor(1:*) !output truncation correction factor, for frequency fitting
double precision cdf(1:K) !output, cdf values at x points
integer :: sList ! error message structure

K <- 100
L <- 2.0
X <- abs(rnorm(K)) + 2*L # rlnorm(K, meanlog = 1.553 , sdlog = 0.117) #
w <- seq(1.0,1.0,length.out=K)
parVal <- c(0.0, 1.0) # mu, sigma - Need INITIAL VALUES
parFlag <- c(0, 0) # correspond to mu, sigma. 0 is the default action. 1 means to not calculate that parameter
stderr <- c(0.0, 0.0)
factor <- c(0.0, 0.0) # Truncation factor
cdf <- seq(0,0,length.out=K)
sList <- 0

callfromr=.C("MleFitTruncLN", as.integer(K), as.double(L), as.double(X), as.double(w), as.double(parVal), as.integer(parFlag), as.double(stderr), as.double(factor), as.double(cdf), as.integer(sList))

#Results
List of 10
$ : int 100
$ : num 2
$ : num [1:100] 4.44 4.24 4.52 4.37 4.97 ...
$ : num [1:100] 1 1 1 1 1 1 1 1 1 1 ...
$ : num [1:2] 1.553 0.117
$ : int [1:2] 0 0
$ : num [1:2] 0.01168 0.00826
$ : num [1:2] 1.00e+02 1.31e-21
$ : num [1:100] 0.297 0.178 0.349 0.25 0.664 ...
$ : int 0

#ECDF plot of original data
xecdf <- ecdf(X)
plot(xecdf)

# Compare with theoretical plot based on parameters found by MLE
xtest <- seq(0,8, length.out=150)
ptest <- plnorm(xtest, meanlog = 1.553 , sdlog = 0.117)
dev.new()
plot(xtest, ptest)

# Compare original and simulated curve by using adk.test
xtest <- rlnorm(150, meanlog = 1.553 , sdlog = 0.117)
#xtest <- rlnorm(150, meanlog = 1.549 , sdlog = 0.112)
adkOut <- adk.test(list(x1=X, x2=xtest))

Fitting Distributions using VGAM::vglm and others
==================================================
To fit a random distribution function, one can use the package VGAM with its vglm function.
An example is:
ylg <- rweibull(5000, 3.2, 1.9)
f3_dist <- VGAM::vglm(formula=ylg ~ 1, family=VGAM::weibull )

In the above example, we create a test random weibull distribution given the parameters (3.2, 1.9)
To use the VGAM::vglm function to find the parameters,
let the formula be Y~1
let the family be VGAM::weibull which is the distribution we want to fit.
Note the output coefficients returned in f3_dist are NOT the parameters.
The parameters are obtained by:
> VGAM::Coef(f3_dist)
shape scale
3.25349663371564 1.90127764909165

> VGAM::summaryvglm(f3_dist)
Call:
VGAM::vglm(formula = ylg ~ 1, family = VGAM::weibull)
Pearson Residuals:
Min 1Q Median 3Q Max
log(shape) -11.108862089551 -0.3482171960797 0.3379254287005 0.6850168947179 0.7955259659881
log(scale) -1.492863009759 -0.7246117432266 -0.2719877869469 0.4750868268133 7.0563972998482
Coefficients:
Value Std. Error t value
(Intercept):1 1.1797303052868 0.011026577908436 106.9897038849
(Intercept):2 0.6425261070716 0.004577189943693 140.3756704388
Number of linear predictors: 2
Names of linear predictors: log(shape), log(scale)
Dispersion Parameter for weibull family: 1
Log-likelihood: -4310.23959834649 on 9998 degrees of freedom
Number of Iterations: 2

Notice that predictors are log(shape), log(scale) and their corresponding values are 1.179, 0.642.
These two values are called the Intercepts because our model is ylg~1
Since they are log, the actual values should be exp(1.179) and exp(0.642) which becomes 3.253 and 1.901.
These last two are consistent with the values from VGAM::Coef which are the parameters for the example
weibull distribution.

Alternative:
library(MASS)
fitdistr(ylg, "weibull")

Fitting poisson with glm:
xSamp <- c(18 15 20 16 29 11 11)
bbb <- glm(xSamp~1, family=poisson())

> summary(bbb)
Call:
glm(formula = xSamp ~ 1, family = poisson())
Deviance Residuals:
Min 1Q Median 3Q Max
-1.588903781006 -1.058919519420 -0.279181657945 0.438723321313 2.603294064982
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 2.841581593727 0.091287051899 31.12798 < 2.22e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for poisson family taken to be 1)
Null deviance: 12.67798701909 on 6 degrees of freedom
Residual deviance: 12.67798701909 on 6 degrees of freedom
AIC: 47.14811314083
Number of Fisher Scoring iterations: 4

To get the fitted Poisson parameter:
1. fitted.values(bbb)
1 2 3 4 5 6 7
17.1428571428641 17.1428571428641 17.1428571428641 17.1428571428641 17.1428571428641 17.1428571428641 17.1428571428641

2. Check with : mean(xSamp)
[1] 17.1428571428571

3. Get value of intercept from summary -> 2.84158
Convert this: exp(2.84158) = 17.1428

Error Handling
===============
Ref: http://mazamascience.com/WorkingWithData/?p=912

The general structure for tryCatch() has the following form:

result = tryCatch({
expr
}, warning = function(w) {
warning-handler-code
}, error = function(e) {
error-handler-code
}, finally {
cleanup-code
}

Alternative is the: try()

Errors:
LoadLibrary failure: %1 is not a valid Win32 application.

Sweave
========
http://www.statistik.lmu.de/~leisch/Sweave/Sweave-manual.pdf
http://www.r-bloggers.com/sweave-tutorial-1-using-sweave-r-and-make-to-generate-a-pdf-of-multiple-choice-questions/
http://www.r-bloggers.com/search/Sweave
http://www.cepe.ethz.ch/education/NPecoHS2010/Sartori-Sweave.pdf

Global variables and scope
===========================
R can handle variables with the same name defined in different scopes.
Example:
aaa <- 2.0 - defined in the interactive environment
go into function func() and print gives
aaa = 2.0
In function func(), define
aaa <- 3.0
Printing gives
aaa = 3.0
Still in func(), rm(aaa) then print(aaa) will give:
aaa=2.0

This behaviour is only valid when aaa is originally a global env variable and the aaa defined in the function is local.
In the function, if aaa is defined as global, eg. aaa <<- 3.0
then this will replace the global definition.

Notes RStudio Server

Notes RStudio Server
=====================

TO DO
Details
Connecting to Linux
Basic Linux Admin
R installations on Linux
R Studio Server Installation
Installing Apache
Configuring Apache Daemon - httpd
R Studio Management / Configuration
Configuring Subversion for Rstudio
Running R-Studio with Subversion
Services That Need To Be Restarted

Details
========
server names, login names, etc --- see details.txt

Connecting to Linux
=====================
Get Open Source SSH software
http://www.bitvise.com/tunnelier-download

To connect via SSH:
1. Open Bitvise Tunnelier
2. Host: <LinuxServer>
Port: 22
Username: <LinuxUser>
Initial Method: password
Password: <>
3. Click on Open New Terminal Console.
4. To check sudo password is working:
sudo more /tmp/dsm.sys.bak
... or use any files that can be viewed by root only.

Basic Linux Admin
===================
To check which version of GCC goes with which version of RH Linux.
http://distrowatch.com/table.php?distribution=redhat

check version of Linux
cat /proc/version
Linux version 2.6.18-308.el5 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Jan 27 17:17:51 EST 2012
uname -a

check 32bit vs 64 bit
uname -m
----- output ------
x86_64
file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped

check packages
yum list installed
yum list available

For X windows:
sudo vi /etc/ssh/ssh_config
- modify this file so that
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

No DISPLAY=myws:0; export DISPLAY
No DISPLAY=127.0.0.1:0.0; export DISPLAY
No DISPLAY=:0.0; export DISPLAY
No DISPLAY=:0; export DISPLAY
No DISPLAY=.0; export DISPLAY

In Remote box, type:
DISPLAY=localhost:10.0; export DISPLAY
xterm &

To avoid setting DISPLAY everytime, edit .bash_profile by adding these lines:
DISPLAY=localhost:10.0
export DISPLAY

Fedora, and perhaps RHEL, rpm packages are "likely" to be installed in:
/lib/
/usr/bin/
/usr/lib/
/usr/share/doc/
/usr/share/man/

top - to see status of processes, press 'q' to quit, press 'H' to toggle threads view.
- press 'n' to enter the number of lines displayed.
mpstat -P ALL - to see stats on all CPUs

/etc/services - a file with list of port numbers and their associated services.
netstat - to see the ports of active services

R installations on Linux
=============================
http://cran.r-project.org/doc/manuals/R-admin.html
http://cran.r-project.org/doc/manuals/R-admin.html#Getting-and-unpacking-the-sources

Manual Downloads:
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R.html
R-2.15.0-1.el5.x86_64 [16 KiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R-core.html
R-core-2.15.0-1.el5.x86_64 [36.0 MiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R-devel.html
R-devel-2.15.0-1.el5.x86_64 [90 KiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/xdg-utils.html
xdg-utils-1.0.2-4.el5.noarch [52 KiB] Changelog by Lubomir Kundrak (2008-01-25):- Fix for CVE-2008-0386 (#429513)

rpm -i R-core-2.15.0-1.el5.x86_64.rpm
warning: R-core-2.15.0-1.el5.x86_64.rpm: Header V4 DSA signature: NOKEY, key ID
217521f6
error: Failed dependencies:
cups is needed by R-core-2.15.0-1.el5.x86_64
libtk8.4.so()(64bit) is needed by R-core-2.15.0-1.el5.x86_64
tetex-latex is needed by R-core-2.15.0-1.el5.x86_64
xdg-utils is needed by R-core-2.15.0-1.el5.x86_64

Type these commands to install dependencies:
sudo yum install cups
sudo yum install tetex
sudo yum install tk.x86_64
sudo rpm -i xdg-utils-1.0.2-4.el5.noarch.rpm
sudo yum install tetex-latex
sudo rpm -i R-core-2.15.0-1.el5.x86_64.rpm

To check rpm's are installed, eg.
sudo rpm -q -i xdg-utils
sudo yum info R-core-2.15.0-1.el5.x86_64 # can yum even if packaged was RPM

To check yum's are installed, eg.
sudo yum info tetex-latex
sudo more /var/log/yum.log

yum configuration details are in:
/etc/yum.conf
/etc/yum.repos.d/*.repo

R Studio Server Installation
==============================
Ref: http://rstudio.org/download/server

1. Extra Packages for Enterprise Linux (EPEL)
Download epel-release-5-4.noarch.rpm from:
http://dl.fedoraproject.org/pub/epel/5/x86_64/

(Transfer that file to the Linux box)

In the Linux box, do (see WARNING below first) the following:
sudo rpm -Uvh epel-release-5-4.noarch.rpm

To check installation:
sudo rpm -q -i epel-release-5-4

WARNING: It may not be necessary to do this step. Not sure what the consequences of avoiding this step is. The main problem with DOING this step, from actual experience, is that it corrupts the yum repository. Actually it added 2 repository which cannot be accessible. The solution is to move the files below to somewhere else:
sudo move /etc/yum.repos.d/epel* <Some Junk Directory>

2. Install R (core or base) package.
Since R is installed in the step above, we just need to check R is installed by:
sudo yum info R-core-2.15.0-1.el5.x86_64 # can yum even if packaged was RPM
OR
sudo rpm -q -i R-core

3. Installing R Studio
Download the Rstudio package from: http://download2.rstudio.org/rstudio-server-0.95.265-x86_64.rpm

Transfer the rpm to the Linux box if not there already.

Install: sudo rpm -Uvh rstudio-server-0.95.265-x86_64.rpm

Check: sudo rstudio-server verify-installation
...the following output from the check still allows Rstudio to run through web in the end - so it is OK.
rserver[20481]: WARNING R include path (/usr/include/R) not found; LOGGED FROM: bool core::r_util::<unnamed>::validateREnvironment(const core::r_util::EnvironmentVars&, const core::FilePath&, std::str
ing*) /root/rstudio/src/cpp/core/r_util/REnvironmentPosix.cpp:379
Starting rstudio-server: [ OK ]

Configuring for External Libraries - WORK IN PROGRESS
- This are R libraries that are not part of the standard R base. They are developed by other users, can be downloaded from the R repository.
- For R studio, let only the svn user have control over the R libraries and they will be stored at:
/home/svn/R/library
- Create the configuration files as follows, edit the file:
sudo vi /etc/rstudio/rserver.conf
- Put this content in the file:
rsession-ld-library-path=/home/svn/R/library
- Test and restart server
sudo rstudio-server test-config
sudo rstudio-server restart

Package Management Proposal
USE CASE 1: A normal R user wants to install packages using Rstudio-server GUI.
- Problem 1: it appears each user need to install their own packages under ~/R/library
- Problem 2: Linux server cannot access outside world.
- Solution:
a. Create a local repository on Linux box, so that Rserver points to http://localhost/src/contrib/
b. On Windows: Manually or write a Web Robot to source all updated packages. Then transfer all packages to Linux server under <Web>/src/contrib
c. On Linux, each user specify the localhost as R repository.
Summary, essentially we run our R Repository mirror. Each user install packages as they need. All users of CBA are guaranteed to use the same version of packageas because they all source it from the same CBA repository.
d. An extra step may be to link all users directories ~/R/library to a common directory on Linux server. This means when one user install a package, it will be available to all users.

USE CASE 2: An R administrator installs packages for everyuser.
The R administrator gets the R library SOURCE, in the form of *.tar.gz and puts in them in <LOCAL_REPO>.
To install individual packages, the R administrator will do:
sudo R CMD INSTALL -l <R_SHARE> <LOCAL_REPO>/adk_1.0-2.tar.gz

... where the example package to install is adk, and the location to install may be <R_SHARE>="/usr/share/R/library"
Check that R_SHARE is listed as one of the directories given by R command: .libPaths()

To remove:
sudo R CMD REMOVE -l <R_SHARE> adk

To Check:
- login as a regular R user and type:
library(help="adk")

With this method, any R user can use the package and all users will be using the same version.
The problem of accessing the repository still remains and has the same solution described as above.

Some R package management commands:
packageStatus()$inst
getOption("repos") - list R repositories
getOption("defaultPackages") - list packages loaded by default
remove.packages(c("pkg1", "pkg2"), lib = file.path("path", "to", "library"))
.libPaths() - default libpaths
"/home/cheec/R/library" "/usr/lib64/R/library"
"/usr/share/R/library" "/usr/lib/rstudio-server/R/library"
Sys.getenv("R_LIBS_USER") - "~/R/library" *** note these are R env vars - they are not Linux env vars
Sys.getenv("R_HOME") - "/usr/lib64/R"
R_HOME/etc/repositories - list of directories
R CMD INSTALL -l <LIB> pk1 pk2 - installs packages pk1, pk2 to location <LIB>
install.packages(c("pk1", "pk2")) - within R, install packages pk1, pk2
install.packages("pk1", dependencies=TRUE) - within R, install packages pk1 and dependencies.
install.packages("~/R/library/adk_1.0-2.tar.gz", repos = NULL)
remove.packages("adk")
library(help="adk") - to check if adk has been installed correctly
update.packages() - checks and updates all necesary packagees
packageStatus()$inst
R CMD check -l ~/R/library ~/R/library/adk_1.0-2.tar.gz - to check and INSTALL the package adk is installed.
sudo R CMD INSTALL -l /usr/share/R/library ~/R/library/adk_1.0-2.tar.gz
sudo R CMD REMOVE -l /usr/share/R/library adk

To set up a local CRAN mirror:
http://cran.r-project.org/src/contrib/
http://cran.r-project.org/mirror-howto.html

http://r.789695.n4.nabble.com/R-2-6-0-packages-installation-through-a-proxy-not-working-td837190.html

Installing Apache
===================
Download http://apr.apache.org/download.cgi
apr1.6
apr-util 1.4.1
apr-iconv1.2.1
Download http://mirror.overthewire.com.au/pub/apache//httpd/httpd-2.4.2.tar.gz

Transfer the *.gz files to the linux box.

Extract the Apache source:
sudo mv <the FOUR *.gz files> /usr/local
cd /usr/local
sudo tar -xzvf <the FOUR *.gz files>

Installing apr:
go to the appropriate directory, eg cd /usr/local/apr
sudo ./configure
sudo make
sudo make install

Installing apr-util
go to the appropriate directory, eg cd /usr/local/apr-util....
sudo ./configure --with-apr=/usr/local/apr
sudo make
sudo make install

Install pcre-devel
sudo yum install pcre-devel.x86_64

Installing apache2
go to the appropriate directory, eg cd /usr/local/httpd...
sudo ./configure --with-apr=/usr/local/apr --enable-proxy --enable-proxy-http --enable-proxy-html --enable-xml2enc
sudo make
sudo make install

Uninstall apache2 - only if you need to uninstall for whatever reasons
go to the appropriate directory, eg cd /usr/local/httpd...
sudo make clean
sudo make distclean

Installing elinks - a text based browser (for checking whether webserver running locally)
sudo yum install elinks

Configuring Apache - to use Reverse Proxy to Rstudio-Server
- apache src files are at: /usr/local/httpd.... --> <SRC_DIR>
- by default, apache is installed under : /usr/local/apache2 (check this by <SRC_DIR>/configure --help)
- from here on, let notation $PREFIX="/usr/local/apache2"
- to start apache server: $PREFIX/bin/apachectl -k start
- to stop apache server: $PREFIX/bin/apachectl -k stop
- to restart apache server: $PREFIX/bin/apachectl restart
- to check if apache server is running: pstree (look for httpd in text)
- to check apache installation, edit the file $PREFIX/htdocs/index.html and write some html content like "Hello World"
open up elinks text browser on the linux box, and goto http://localhost, you should see the index.html modified above.
open up normal browser on your PC desktop within CBA network, and goto http://<HOST>, you should see the index.html modified above.

Configuring Proxies
References
- http://www.apachetutor.org/admin/reverseproxies
- http://thelowedown.wordpress.com/2008/10/12/reverse-proxy-with-apache/
- http://code.google.com/p/puppeteer/wiki/ProxyConfiguration
- http://wiki.apache.org/httpd/ProxyAbuse

Put the following code into PREFIX/conf/httpd.conf file:
<VirtualHost *:80>
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass / http://localhost:8787/
ProxyPassReverse / http://localhost:8787/
</VirtualHost>

- stop apache server (see above)
- start apache server (see above)
- Open a browser on your local PC desktop and goto http://<HOST> (no port numbers needed). You will be REDIRECTED to R-Studio.

Configuring Apache Daemon - httpd
===================================
Note there are TWO versions of apache. One is the RPM package that came prebuilt with RHEL. The other one is the one manually built using the method above. The self-build one is needed over the RPM one because several specific options for Rstudio requires this.

The RPM apache is configured to be ready to start up with the following configuration.
File: /etc/rc.d/init.d/httpd - startup script that has the following relevant information
- points to /etc/sysconfig/httpd - has httpd system related information
- points to /usr/sbin/apachectl - control program
- points to /usr/sbin/httpd - actual daemon program
- points to /var/run/httpd.pid
- points to /var/lock/subsys/httpd
- points to /etc/httpd/conf/httpd.conf - main server configuration file

For the apache build locally, it is not set to startup, but has the following files
- /usr/local/apache2/bin/apachectl - control program
- /usr/local/apache2/bin/httpd - actual daemon program
- /usr/local/apache2/conf/httpd.conf - main server configuration file

*** You will be SUPPLIED / GIVEN a file called httpd_start
The idea is to modify the already existing /etc/rc.d/init.d/httpd (from RHEL's Apache rpm) so that it has the contents of the given httpd_start file
1. Edit the file /etc/rc.d/init.d/httpd so that its contents are identical to httpd_start.
OR
2.Alternatively to copy-paste is to upload httpd_start to directory /etc/rc.d/init.d/ to REPLACE existing httpd file.
3.Also do this to hide another file: sudo mv /usr/sbin/httpd /usr/sbin/httpd_REDHAT_old

to check that this new httpd is working, do the following:
cd /etc/rc.d/init.d
sudo ./httpd start
--- check that the webserver is running ----
sudo ./httpd stop

Then do the following:
sudo chmod 755 /etc/rc.d/init.d/httpd - to ensure it has permissions to start
sudo chkconfig --list - do this to check that httpd is registered or not
sudo chkconfig --add /etc/rc.d/init.d/httpd - to register for startup, if not there yet
sudo chkconfig --level 2345 httpd on - to switch on at specific runlevels

Ref - for init.d/httpd startup script
http://www.techiecorner.com/104/how-to-auto-start-apache-during-boot-time-linux/
http://www.zrinity.com/developers/apache/apache2src.cfm

R Studio Management / Configuration
====================================
To manually stop, start, and restart the server you use the following commands:
sudo rstudio-server stop
sudo rstudio-server start
sudo rstudio-server restart

To list all currently active sessions:
sudo rstudio-server active-sessions
To suspend an individual session:
sudo rstudio-server suspend-session <pid>
To suspend all running sessions:
sudo rstudio-server suspend-all
The suspend commands also have a "force" variation which will send an interrupt to to the session to request the termination of any running R command:
sudo rstudio-server force-suspend-session <pid>
sudo rstudio-server force-suspend-all
The force-suspend-all command should be issued immediately prior to any reboot so as to preserve the data and state of active R sessions accross the restart.

Taking the Server Offline - If you need to perform system maintenance and want users to receive a friendly message indicating the server is offline you can issue the following command:
sudo rstudio-server offline
sudo rstudio-server online
These two commands are independent of start stop. When suspended, even if the server is restarted, rstudio is still not accessible (ie offline), until it is switched back to online again.

Workflow for taking R offline:
Method A
- on server: sudo rstudio-server offline
- on user PC: 1. "Error: Status code 503 returned" display in the R session 2. "Rstudio Temporarily Offline" dialog appears. 3. User cannot do anything.
- on server, this has to be done IN ORDER, otherwise session will not start.
1. sudo rstudio-server restart
2. sudo rstudio-server online

Version Control using Subversion
===================================
RHEL5 has got subversin 1.6.11 pre-installed for both i386 and x86_64
To use Subversion with R, the work needs to be organized a Rstudio Projects.

1. To check if the svnserver starts automatically, whether svn is preinstalled for Red Hat, or self install later, type:
ls -laF /etc/init.d/svnserve
to see if the file exists. This file runs a script to start the daemon.

2. Create svn user and group, type:
sudo useradd svn - use id svn to check user details
sudo passwd svn - make password for svn -> "svn"
sudo usermod -a -G R_POC_TEAM svn - this puts the user "svn" to the group called R_POC_TEAM.

Ensure that all other user of svn is in the same group called "R_POC_TEAM". To check which groups the users are in, type:
sudo more /etc/passwd

3. Create the SVN repository for all members of Group Quantitative Analytics (GQA).
- this is done only ONCE, when a new VM and subversion is installed.
- all users and the 'svn' user need to be in the same group. In this example, all these users belong to a group called R_POC_TEAM. If this is not true, make it so.
- Login as svn user.
- create the repository called QuantAnalytics, type:
svnadmin create --fs-type fsfs /home/svn/QuantAnalytics

This creates the repository QuantAnalytics, that will contain all projects. Note that --fs-type fsfs is the preferred filesystem for svn rather than the Berkley DB filesystem (see manual).

4. Check the file /etc/services. If it does not contain the following lines, then add them:
svn 3690/tcp # Subversion
svn 3690/udp # Subversion

5. Configuring Authentication and Authorization
- LOGIN AS USER called 'svn'
- go to the QuantAnalytics repository, eg:
cd /home/svn/QuantAnalytics
- Create a file under ...QuantAnalytics/conf/svnserve.conf, with the following content:
[general]
password=db = passwd
realm = QA realm
anon-access = read
auth-access = write
...... passwd is actually the word "passwd".

- Ensure the file called 'passwd' exist under ...QuantAnalytics/conf/ and that it has the following content:
[users]
harry = harryssecret
sally = sallyssecret

.... where harry and sally are valid users in the linux box.

6. Hide the absolute path of the svn server and provide relative paths to user only.
Edit this file: /etc/rc.d/init.d/svnserve

by adding "-r /home/svn" to the following line if it exist, so that it becomes:
args="--daemon --pid-file=${pidfile} $OPTIONS -r /home/svn"

Now when the svn server is mentioned, it is done by:
<svn server>/QuantAnalytics instead of <svn server>/home/svn/QuantAnalytics

The following information are for references:
- Access SVN. The URL pointing to the repository is:
svn://<HOST>/QuantAnalytics
svn://localhost/QuantAnalytics

- Some svn admin commands: svnadmin, svnlook, svndump, svndumpfilter, svnsync

- svnlook info <repos> - prints information for the repository at <repos>.
The information include author, date, number of lines of log, log message

- sudo killall svnserve - force termination of any active svnserve services. Also solves the error with message:
" svnserve: Can't bind server socket: Address already in use"

- sudo /etc/init.d/svnserve start - to start svn server once off.

***** How to setup iptables - NOT Needed here
http://articles.slicehost.com/2007/9/5/introduction-to-svnserve

Running R-Studio with Subversion
==================================
WARNINGS
A. NEVER NEVER NEVER have filenames with SPACES or ANY OTHER symbols except . (fullstop) and _ (underscore).

1. To run Rstudio, open a browser and type this URL:
http://<HOST>

To login as another user (for administrators only), when the user name field is no longer visible, type:
http://<HOST>/auth-sign-in

The sign in password is the same password of the Unix account.

2. Go to Tools - Options, click the Version Control icon on the left navigation pane.
- Under "SVN executable:", type: /usr/bin/svn

3. The Repository URL is:
svn://localhost/QuantAnalytics

4. To Finish R, there are two ways:
- Within Rstudio, click File - Quit R.
This will finish off the current R but immediately asks you if you want to start a new R version.
- On the top right, click Sign-Out
This will sign you out, but your session is saved on the server. When you login again, you will see the same session where you left off, including all the variables in the workspace will still be there.
- Also while working in a project, MUST close project before exiting R, otherwise the same project opens automatically when Rstudio is started. To close project, click: Projects - Close Projects.

5. (Recommended) To create a new project and at the same time, enabling version control.
This is the recommended way, even if you have existing R code files and existing project, to simplify the process of putting your code into version control, just create a new project associated with version control. Then copy your files into this new project.
- Once logged in to Rstudio, click Tools - Shell
- Type: mkdir ~/<ProjName>
where ~ means your home directory, <ProjName> is the name of your project
- Type: svn import ~/<ProjName> svn://localhost/QuantAnalytics/<ProjName>
- Click Close, to get back to main Rstudio
- Click Project - New Project
- Choose "Version Control" from the New Project dialog.
- Click "Subversion", not "Git"
- Enter the following details:
Repository URL: svn://localhost/QuantAnalytics/<ProjName>
Username: <your Linux account username>
Project Directory Name: <ProjName>
Create project as subdirectory of: ~
- Click "Create Project"
- RStudio will open up in the new project directory. Experiment a bit here. Create a file, write something, save it.
- Copy files from other directories into this directory. Then add these files to version control.

6.A. When a Directory for a group of R files exist, to put it in version control, first formalize that directory as a R project, then add to version control.
- Once logged in to Rstudio, click Project - New Project
- Choose "Existing Directory" from the New Project dialog.
- In the Project working Directory, enter the existing directory path, eg. ~/ExistingDir
- Click "Create Project".
At the end of this stage, a new project has been created to encompass old files in an old directory.

6.B. To put existing R project directory into SVN:
- Login to Rstudio and open the project created in the previous step.
- Check that Rstudio is now in that project directory, type: getwd()
- On the Menu, click Tools - Shells
- To upload files to SVN, type:
(In General) svn import <local path> <SVN repo URL> -m "your message"
(Example) svn import ~/myProj svn://localhost/QuantAnalytics/myProj -m "Initial Import"

svn delete svn://localhost/QuantAnalytics/myProj/.Rproj.user
... then press C
svn delete svn://localhost/QuantAnalytics/myProj/.Rhistory
... then press C
The last two lines are to remove your personal R configuration files - since these should not be in the server.
Alternatively, to avoid the "svn delete", we can just selectively import the R source files we intend to store.

svn checkout --force svn://localhost/QuantAnalytics/myProj ~/myProj
svn update
svn commit

- Close the Shell from Rstudio.
- Now back in the main Rstudio interface, Close the R project
- Reopen the R project. You will now find in the top-right Box, next to the "Workspace" and "History" tabs, there is a new "SVN" tab. This "SVN" will track any changes made to files which are under version control.

7. A few other tasks with SVN and R studio.

To UN-version control your local directory - the situation is directory A is in version control, and directory A is in your local workspace already. You wish to keep the contents in your workspace. You wish to break any links between your workspace and version control.
- go to directory A
- type: rm -r -f .svn

To delete any path or branch from version control. Warning: Deleting a path and files from version control may be deleting some files your colleagues are using. DO NOT do this until you check with them. Danger: this command may allow you to accidentally delete the wrong path in version control, ie. deleting your colleauges files. BE SURE what you are deleting.
- svn delete svn://localhost/QuantAnalytics/<your directory>

NEVER TYPE:
svn delete svn://localhost/QuantAnalytics

Services That Need To Be Restarted
===================================

Services are: Apache Rstudio SVN
<ServiceNames>: httpd rstudio-server svnserve
Runlevel default: NA 2345 NA
Runlevel default: 2345 2345 2345

Runlevel default is the out-of-box startup configuration before any changes have been made.
This is obtained from : chkconfig --list

Do the following to register httpd and svnserve will start automatically
sudo chkconfig --level 2345 httpd on
sudo chkconfig --level 2345 svnserve on

Redhat Enterprise Linux runlevels are (according to /etc/inittab)
# Default runlevel. The runlevels used by RHS are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#

Checking if daemon has started:
service rstudio-server status
chkconfig --list
sudo more /etc/inittab - shows how the system is initialized
sudo ls /etc/rc.d/init.d/ - list of daemon files, used or not used in startup
ls -latrF /etc/rc.d/rc3.d/ - links to actual files in init.d, at this particular runlevel 3

Tuesday, May 15, 2012

Tribute to Nikola Tesla

Here is a very descriptive tribute and in honour to Nikola Tesla - the greatest Geek that ever lived.

http://theoatmeal.com/comics/tesla

Warning: That article contains unpleasant descriptions of Edison - eg "douchebag"

Tuesday, May 01, 2012

Development - Cloud Platform As A Service (PaaS)

This post is about the various services available on the cloud for software development. In particular this type of cloud service is called Platform As A Service (PaaS). Typical example of his are:
1. VMware's Cloud Foundry http://docs.cloudfoundry.com/
2. Microsoft's Azure Cloud http://www.windowsazure.com/en-us/

Difference between Platform (PaaS) and Infrastructure (IaaS) cloud services
The PaaS are different to Infrastructure as a Service (IaaS) type of cloud, example being Amazon's Cloud Services like EC2 (Elastic Cloud). IaaS requires the user to setup all the infrastructure. As a developer or programmer, that means you need to install your own tools, compilers, IDE, development framework, etc. PaaS on the other hand has the development framework already installed and configured. As a developer, you just need to write code according to the framework and upload it to the cloud to run.

Cloud Foundry
Cost: Free - Open Source Platform
Infrastructure - http://docs.cloudfoundry.com/infrastructure/overview.html
Framework - http://docs.cloudfoundry.com/infrastructure/overview.html Click on the Framework menu item. Currently it supports Java, Node.js, Ruby environments.

Windows Azure
Cost: Paid - http://www.windowsazure.com/en-us/pricing/details/
Framework - Apparently it is NOT limited to the .Net development environment. The full features are at:
http://www.windowsazure.com/en-us/home/features/overview/

Links - Big Data, Hadoop

This post is a collection on the topic of Big Data and its currently famous tool called Hadoop. Feel free to comments and recommend useful links.

What is Big Data?

Src: http://online.wsj.com/article/SB10001424052702304723304577365700368073674.html
"Big Data refers to the idea that an enterprise can mine all the data it collects right across its operations to unlock golden nuggets of business intelligence. And whereas companies in the past have had to rely on sampling, Big Data, or so the promise goes, means you can use your entire corpus of digitized corporate knowledge. It is, by all accounts, the next big thing."

Who do we need for Big Data?
http://gizmodo.com/5906204/the-problem-with-big-data-is-that-nobody-understands-it

The Problem With Big Data Is That Nobody Understands It
"They can take a data set and model it mathematically and understand the math required to build those models; they can actually do that, which means they have the engineering skills…and finally they are someone who can find insights and tell stories from their data. That means asking the right questions, and that is usually the hardest piece."

Sunday, April 22, 2012

How Does a Router Protect

I posed the question to myself of "How does a router protect" based on my curiosity on what something I remembered that a router is already acting as a firewall. So I googled on the topic and found some very interesting results. The google results are listed below. But first, my summary is this:

1. Router uses NAT to protect computers connected behind the router.
2. The Router NAT technique protects outside attempts to talk to the computers behind the router - ie blocks inbound traffic.
3. But Router NAT does not by default block outbound traffic. A computer already infected may call outside to its base and get information or commands back.
4. Some say router is enough, other say software firewall is necessary.
5. Router NAT does not protect against computers or configuration which uses:
- VPN - to connect into a company's network from home securely.
- Port Forwarding - required when running a web server.
- DMZ - used by gamers sometimes to enable playing network games.
Any of these will by pass the NAT protection mechanism and expose the home computer and others on the network.

Setting up Cascading Router (LAN to LAN or WAN to LAN)
http://www.linksys.com/au/support-article?articleNum=132275

Using A Modem and Router combination
Internet -> ADSL Modem -> Router -> Device

(WAN stands for Wide Area Network and is the IP address given to you by the Internet service provider)

Double Nat
http://www.howtogeek.com/255206/how-use-your-router-and-isps-modemrouter-combo-in-tandem/
To overcome the Double NAT problem, one way is to use Bridging.
Switch the Modem into Bridge mode.
"Bridging is simply an old networking technique that transparently links two different networks."
Consequences:
- the modem will become a modem only, with have no effective routing functions.
- no devices can be connected directly to the modem unit
- no devices cab be wirelessly connected directly to the modem unit

https://www.cnet.com/how-to/home-networking-explained-part-8-cable-modem-shopping-tips/
On the other hand, it's a little bit more work to add a Wi-Fi router to your existing gateway.
1.you need to connect the new router's WAN (or Internet) port to the gateway.
2.make sure that the new router has a different IP address from that of the gateway. (Chances are they are already different, but if not, you will need to change that of the new router before plugging it to the gateway.)
3. And finally, apart from turning off the Wi-Fi network of the old gateway, if you want the new router to get the WAN IP address, you will need to configure the gateway to pass that to the router. The means of doing this varies depending on the gateway itself. The passing of the WAN IP address is only necessary if you want to set up customized Internet-related services, such as those mentioned in Part 9 of this series.

The Ultimate Modem/Router Setup Thread
http://www.tomshardware.com/forum/33700-42-ultimate-modem-router-setup-thread

When is an NAT router inadequate protection?
http://www.dslreports.com/faq/9787
temporary mirrored at:
http://xtechnotes.blogspot.com.au/2012/04/when-is-nat-router-inadequate.html

How Does A Router Protect My Computer?
http://www.askageek.com/2006/10/17/how-does-a-router-protect-my-computer/

A Router Can Protect your Computer
http://www.compukiss.com/articles/a-router-can-protect-your-computer.html

To what extent does the firewall on a router protect you?
http://askville.amazon.com/extent-firewall-router-protect/AnswerViewer.do?requestId=747083

How does a router protect you?
http://forums.cnet.com/7726-6035_102-5152551.html

Does my router have a firewall or not?
http://ask-leo.com/does_my_router_have_a_firewall_or_not.html

How do I protect users on my network from each other?
http://ask-leo.com/how_do_i_protect_users_on_my_network_from_each_other.html
info on dual router layer / double NATing architecture.

When is an NAT router inadequate protection

This article is mirroring the article at : http://www.dslreports.com/faq/9787 . The dslreports.com website seems to be down, and has no expected online time.

24 Apr 2012 - The original site seems to be working again. So go to http://www.dslreports.com/faq/9787

The main points of the article is extracted here:

----------------------------------------------------------------------

When is an NAT router inadequate protection?

1. Depending on your network configuration, an NAT router can be a very cost-effective, inexpensive and reliable addition to your computer's security. At US$40 to $70, they can be worth getting even if you only have one computer.

1.1 You should definitely run a software firewall on any computer that connects to AOL using a different Internet Service Provider (AOL's Bring-Your-Own-Access plan or AOL MAX using an ISP) no matter what kind hardware firewall or NAT router you have.

1.2 If you have to turn on port forwarding or the DMZ to run servers or other applications you should consider either a software firewall or a more expensive SPI firewall.

1.3 Generally software firewalls provide valuable additional protection that supplements the protection provided by NAT routers and SPI firewalls.

Ideally a software firewall should be an additional layer of protection behind an NAT router or external firewall. For homes a free version of a software firewall is normally adequate for this additional layer of protection.

- ZoneAlarm Free
»www.zonelabs.com/store/content/home.jsp
Look for the free version / free download, and continue to ask for it rather than the Pro version.

- Sygate Personal Firewall
»download.com.com/3000-2092-10049···g=button

- Kerio Personal Firewall Limited Free Version (Sunbelt Kerio Personal Firewall)
»www.kerio.com/kpf_download.html
Look for the "limited free" version.

For businesses, computers running public servers, and computers on wireless networks, a paid-for version of a software firewall provides more protection by allowing more customization and more precise control.

2. In selecting an NAT router, software firewall, or hardware firewall, consider its logging and alerts capabilities.

3. If the router or firewall is wireless, secure the wireless interface.

4. Firewalls are not a replacement for adequate backups of data. (Firewalls don't protect against real fires, or burglars.) /faq/10194

5. Other security precautions still need to be taken. For example, operating systems and anti-virus software need to be properly installed, configured and updated.

6. There is no hardware or software you can install that will protect against massive amounts of traffic jamming your communications lines. "SPI firewalls" only protect against certain types of denial of service (DoS) attacks involving malformed packets, or protocol sequence violations and vulnerable software.

7. Historically, the original network firewalls did not do packet inspection. They were rule based, using tables of permitted IP addresses and ports. Packet inspection is not historically in the definition of firewalls.

8. The NAT firewall was a major advance. It limited inbound traffic based on the basic state of communications with the external IP address. Outbound traffic triggered permission for inbound traffic.

9. This is basically how a pure many:1 NAT router works. M:1 is the kind of router commonly used for home and SOHO users to provide a connection for many local computers using one public IP address.

10. Port forwarding bypasses the state table and that source of protection provided by the NAT router. Port forwarding (on a pure NAT router) causes almost all traffic that arrives at a particular port to go to a particular local IP address. (Basic packet filtering is the only protection for the port.)

11. The DMZ should be totally avoided on most NAT routers.

A DMZ is not normally required, provided you know your software. Check the software vendor's website, or email their support area, or search here in BBR, to find out what ports you need to set as trigger ports for which ports, or which ports to forward.

If you really do need a DMZ, use a device that treats the computer in the DMZ as though it was an untrusted computer outside your local network. Ordinary NAT routers do not normally provide this type of DMZ; they normally just forward all unsolicited traffic to the machine in the DMZ, leaving it with no NAT protection.

Here are some security testing sites: /faq/5503

Here is more on securing your home computer: /faq/8463

Here is more on securing a wireless router: /faq/8698

For discussion about your individual circumstances you can post a message in the BBR Security Forum here: /forum/security

Friday, April 20, 2012

Security - Phishing examples

This post is a collection of sample phishing emails. Of course there are plenty of variety, I'm just listing the ones here which caught my attention. Feel free to post in the comments any examples of phishing you have encountered.

Case1:
The payload itself is delivered as a zip file which this email tempts the user to open. Obviously the file is not attached here. The point is that the email below looks very very legitimate.

From: HALL GILL [mailto:pilotstation@computerpostage.com]
Sent: Thursday, 19 April 2012 9:26 PM
To: xxxxxxxxxx
Subject: An error at the delivery

Delivery information,
Your parcel can’t be delivered by courier service.
Status deny: Address delivery doesn’t exist in database.
LOCATION OF YOUR PARCEL:Tempe
STATUS: sort order
SERVICE: Expedited Shipping
NUMBER OF YOUR PARCEL:U707019275 NU
INSURANCE: Yes
Postal label is enclosed to the letter.
Print your label and show it in the nearest post office of USPS
Important information! If the parcel isn’t received within 30 working days our company will have the right to claim compensation from you for it's keeping in the amount of $5.64 for each day of keeping.
You can find the information about the procedure and conditions of parcels keeping in the nearest office.
Thank you for attention.
USPS Global Services.

Thursday, April 12, 2012

Introduction to Cloud Computing for Newbies | Cloud Computing Journal

Saturday, March 31, 2012

Magic Squares - Explained

Magic Squares - explained
=========================

Here is a 3 x 3 magic square.

- the rows add up to T = 15.
- the columns add up to T
- the diagonals add up to T

Here's how to do it:

x 1 x ----- x 1 x ----- x 1 x
x x x ----- x x x ----- 3 x x
x x x ----- x x 2 ----- x x 2

x 1 x ----- x 1 x ----- x 1 6
3 x x ----- 3 5 x ----- 3 5 x
4 x 2 ----- 4 x 2 ----- 4 x 2

x 1 6 ----- 8 1 6 ----- 8 1 6
3 5 7 ----- 3 5 7 ----- 3 5 7
4 x 2 ----- 4 x 2 ----- 4 9 2

1. Start from top row middle column with S=1, S is the starting value.
2. Add 2,3,4,5,6,7,8,9 diagonally, that means try to go UP then RGIHT.
3. If UP is blocked, go to Right column, and fill the bottom row. eg. like filling "2".
4. If RIGHT is blocked, go to the LEFT most column of one row above, eg like filling "3".
5. Since this is a 3x3 square, after 3 numbers, then fill the next number just one row DOWN.
eg like filling in "4".
6. Repeat from step 3.

To generalize to N x N magic squares (MS), where N is ODD,
- in step 5 above, after filling every N numbers, move down exactly ONE row.
- the movement of UP -> RIGHT, that is diagonally, is same for all N x N magic squares.
- the relationship between the starting number S and the total number T is like this.
for 3x3 MS: T / 3 - 4 = S , 3 x 1 + 1 = 4
for 5x5 MS: T / 5 - 12 = S , 5 x 2 + 2 = 12
for 7x7 MS: T / 7 - 24 = S , 7 x 3 + 3 = 24
for 9x9 MS: T / 9 - 40 = S , 9 x 4 + 4 = 40
for NxN MS: T / N - K = S , N x (N-1)/2 + (N-1)/2 = N*N/2 - 1/2

- The sum of all rows or columns or diagonal T, must be divisible by N.
- For any N, the minumum S is 1.
- For any N, the constant K = N*N/2 - 1/2
- For any N, the minimum T is when S = 1,
so Tmin = (1 + N*N/2 - 1/2 ) * N

Sunday, March 04, 2012

How to Boost Your Broadband Speed

Here are 10 very simple tips to increase broadband speeds and links to several useful tools.

1. Test Your TV
Apart from TV there are other electrical equipment that can cause interference.

2. Avoid Extension Wiring - longer wire has poorer signals

3. Tweak your Wifi -
NetStumbler software (www.netstumbler.com/downloads) - use to which WiFi channels your neighbours are using.

4. Get Diagnosed - get help from your ISP.

5. Sort your system
Use unblocka (www.unblocka.com) to tune your settings to improve speed.

6. Keep your router up to date

7. Fine Tune MTU settings
MTU is one of the modem settings that can be changed. It controls the packet size of transmission. To identify issues with MTU - see www.pcauthority.com.au/links/129broad1. A guide on how to adjust MTU is found in www.dslreports.com/tweaks/MTU . Also the Speed Guide TCP Optimizer (www.speedguide.net/downloads.php) will identify the optimal MTU.

8. Ask for Interleaving
If you cannot solve your interference problems, you may be able to ask your ISP to switch on interleaving to improve performance. This essentially chops the packet into smaller pieces and has better error correction. This improve stability but may increase or decrease speed.

9. Replace your filters
A guide to test filters can be found at: www.pcauthority.com.au/links/129broad2

The technical details of what the inside of a filter looks like can be found in (www.adslnation.com/support/filters.php)

10. Watch out for AR7 routers
Routers with the AR7 chipset is known to have a fault so may cause connection to drop out. To find a list of router models containing this chipset, go to www.linux-mips.org/wiki/AR7
Firmware updates for some routers to correct this issue may be listed in www.pcauthority.com.au/links/129broad3

Tuesday, February 28, 2012

Security - Manage Privacy with Google

Google is implementing significant changes to its privacy policy from 1 March 2012. Most users would have received notification of this.

Here's a news article discussing it:

If you use Google, you may want to read this
Stephen Hutcheon, February 29, 2012 - 7:33AM
How your web history page should look after you've clicked "remove".
Opinion: Australia absent in Google privacy feud
"Today is your last chance to adjust your Google privacy settings ahead of a major change to the way Google collects and collates data about you, its users.
From March 1, the company will begin to aggregate all the information it acquires about its users who are logged in to Google services into a single, unified pool of data."

Here is a list of tools on Google's page to help address privacy concerns:
http://www.google.com/intl/en/privacy/tools.html

Here are a few interesting items:

1. Use Google Search in Encrypted form: https://encrypted.google.com

2. Block Third party cookies and site Data.
For Chrome Browser:
a. Go to Options - Under the Botnet - click Content Settings - select "Block third-party cookies and site data".
b. On the same page, click on Manage Exceptions, to create a white list of trusted sites where cookies are allowed.
c. On the same page, click All Cookies and Site Data, and check which cookies are stored.

3. Instructions on how to transfer content in and out of Google products such as Google Docs and more:
http://www.dataliberation.org/

Sunday, February 19, 2012

Tips: Windows Shortcuts

Windows Shortcut Keys
F2 - rename a file
F3 - Find - opens up a dialog to search for word.
F6 - Go to Address Bar in most browsers
Shift + Right-Click - displays the context menu
Windows + B - select the first app in the task bar
Windows + E - opens Windows Explorer
Windows + F - opens the Windows file search window
Windows + L - locks your Windows immediately
Windows + M - clears the desktop, ie. minimizes all windows
Windows + R - opens the Run dialog
Windows + F1 - opens help for Windows
Windows + Tab - Switch between programs. Add the Shift key to go backwards
Windows + Pause/Break - Opens the System Properties dialog
Shift + Delete - to delete a file permanently
Ctrl + Drag - copy or moving files
Ctrl + Enter - In a browser, type the name of the website without www and com, then press these keys to go to the website
Alt + Esc - put this window to the back of the list
Alt + Ctrl + Fullstop - turn a fullstop into ellipsis.
Alt + F4 - Exits the current application, or exit Windows if there is no opened application
Alt + PrntScrn - Screen capture just the current window, not the whole desktop

Updated 27 Oct 2012
The following shortcuts are mainly for Windows 8 Metro

B Move focus to notification tray
C Show Charms menu
D Show Windows desktop
E Launch Windows Explorer
F Show Metro File Search screen
G Cycle through desktop Gadgets
H Show Metro Share panel
I Show Metro Settings panel
J Switches focus between snapped Metro applications
K Show the Devices panel
L Lock PC
M Minimise all Windows on the desktop
O Lock device orientation
P Choose between available displays (Projector)
Q Show Metro Search screen
R Show Run Dialogs
T Cycle through Taskbar icons
U Show Ease of Access Centre
V Cycle through toast notifications
W Show Metro Settings Search panel
X Show Power User Commands or Mobility Centre
Z Show the App Bar

1-9 Show/Launch Application from Taskbar
Page Up / Down Moves tiles to the left/right
Tab Switch between applications
, (comma) Aero Peek (desktop)
. (full stop) Snap Metro style app to right side of the screen
Shift . (full stop) Snap Metro style app to the left side of the screen
Space Switch input language and keyboard layout
Enter Launch Narrator
Arrow keys Aero Snap (desktop)

Updated 2 Nov 2012

http://pogue.blogs.nytimes.com/2012/10/25/a-windows-8-cheat-sheet/
Here are some instructions on shortcuts and other keys for the new Windows 8 interface.

Thursday, February 16, 2012

How to Program

The following section discusses the use of Linux/Unix as a programming environment. Below is a list of some tools. More details are covered in the linked article.

Unix as IDE: Introduction
http://blog.sanctum.geek.nz/series/unix-as-ide/

File and project management — ls, find, grep/ack, bash
Text editor and editing tools — vim, awk, sort, column
Compiler and/or interpreter — gcc, perl
Build tools — make
Debugger — gdb, valgrind, ltrace, lsof, pmap
Version control — diff, patch, svn, git

Some highly recognized books on programming:

The Little Schemer - Daniel P. Friedman
Mastering Algorithms with C - Kyle Loudon
C Programming Language (2nd Edition) - Brian W. Kernighan, Dennis M. Ritchie

Thursday, February 09, 2012

How To Recover Missing Systray Icons

This is still Work In Progress, but here are a few sites about how to solve the Missing Icon In Systray Bug.

https://www.google.com.au/search?ix=hca&sourceid=chrome&ie=UTF-8&q=%22missing+icons%22+in+taskbar&oei=iCIzT4LLKsybiQey9vTrAQ

Google: "missing icons" in taskbar

http://winhlp.com/node/16

http://www.techspot.com/vb/topic123221.html

http://www.techspot.com/vb/topic42925.html

http://en.kioskea.net/forum/affich-31257-how-to-recover-lost-missing-system-tray-icon

http://en.kioskea.net/forum/affich-23085-restore-missing-icons-start-menu

http://www.annoyances.org/exec/forum/winxp/t1034225301

Wednesday, February 08, 2012

Privacy - How To Cover Your Web Surfing Tracks and Internet Presence

When we go Web Surfing or Browsing the Internet with a web browser, we may think it looks like:

                Me ->  Website

but in fact it is more like

               Me ->  My Internet Service Provider (ISP) -> Website

When a web page appears on your computer, there is no magic - information is passed to and from Me, ISP and Website. Basically, anything about you that can be known, your IP (internet address), location, name you sign up with, etc, can be obtained by ISP and Website and by others.

There are many valid and legal reasons why people would like to keep their privacy from their ISP and the Websites they visit. So there are ways to Not Reveal to your ISP which websites you visit, and Not Reveal to the Websites where your real location is.

Using VPN

Hotspot Shield - This software connects your computer via a VPN to the servers of HotSpot Shield which then visits the Websites. It looks like this:

               Me --->           My ISP              HotSpot Shield Servers -> Website
                              ----> VPN Tunnel ---->

So the Website thinks the visitor came from HotSpot Shield servers, instead of coming from you.
The ISP sees encrypted data and does not know which Website you are requesting to see.

Drawback: Theoretically your information is still known by HotSpot Shield servers. Well you have to trust someone in the end......

Here is a guide to set up VPN in Windows 7
http://www.pcworld.com/article/210562/how_set_up_vpn_in_windows_7.html

VPN Gate - An experimental VPN setup in Japan. Free to join.
http://www.vpngate.net/en/

Using Proxies

From a very high level, proxies work in a similar way to VPN. The key difference is that the traffics is not necessarily encrypted. However, the request to a website is send via another server in the middle, so that the destination website does not know it came from you. Since our information passes through a web proxies, we must trust those web-proxies that we use.

Here are some web-based proxies:
- Proxify - proxify.com

- Anonymouse - anonymouse.org
- Hide My Ass - www.hidemyass.com

There are manual proxy servers which require your browser to be configured to make use of the web proxy. The following sites maintain a list of proxy servers:
- Proxynova - www.proxynova.com
- Hide My Ass - www.hidemyass.com

Other Ways

There are many other ways to cover your tracks. This site http://www.how-to-hide-ip.info/hide-ip-tools/ has a collection of tools that cover a variety of ways, such as:

Tools on your PC for hiding IP
Using Proxy in your Web Browser
Proxy Sites
Proxy Lists
IP checking sites
VPN services

Covering Web Browsing activities: using Tor www.torproject.org

In fact TOR (The Onion Router) is an ongoing and mature project with a lot of features that enable anonymity on the internet. TOR comes with a Bundled Browser with TOR preconfigured. TOR also has something for smart phones (see below).

For Smartphone, Mobile, Android systems:

Orbot - this is based on the Tor technology but available to Android on smartphones.

Email Anonymity
Use the following to protect mask your identity when sending emails:
- Anonymouse - anonymouse.org
- Hide My Ass - www.hidemyass.com

When you go to any these sites, your browser security may flag some of these sites to be suspicious. This could be due to their nature to hide identities or the fact that they are suspicious. In any case, make your own decision before visiting these sites.

Thursday, January 19, 2012

How To Clean Up Windows

This article prevents some ideas of how to clean up your computer which has the Windows Operating System. The clean up here does to refer to any kind of virus or spyware removal. Instead, this article focuses on how to get rid of the junk that we or Windows itself accumulates on your computer throughout the years.

1. Scan using antivirus.
Although this article is not about removing virus but on the cleaning of junk files, one of the first step is to simply do a virus scan - just in case. Some useful articles are:
- using online virus scanners: http://xtechnotes.blogspot.com/2008/07/antivirus-online-scan.html
- discounted antivirus software http://xtechnotes.blogspot.com/2012/01/security-software-discounts-and-special.html
- news on security http://xtechnotes.blogspot.com/2011/08/news-security.html
- how to secure your computer: http://xtechnotes.blogspot.com/2010/03/how-to-secure-your-computer.html

2. Remove all unwanted programs.
Find a list of all your installed programs by going to Control Panel - Add or Remove Programs.
Decide which programs you do not need and uninstall them.
- some uninstaller software: http://xtechnotes.blogspot.com/2011/07/links-to-free-software.html

3. Find what is running in the background
Windows services are programs that run in the background after starting up themselves when the computer is switched on. Many will not tell you they are running. To find a list of these "services", go to Control Panel - Administrative Tools - Services.
Find those services which you know is definitely not needed, whether they are from Windows or not. There is no clear way to identify which service you don't want - they will come from experience. Basically just look at the name of the services - a weird name does not mean it is not needed. Sometimes look for a service with a simple name which you definitely know is not required. Example: If your computer has no wireless connection, then look for service with the name wireless and Disable it.

4. Clean the Registry
WARNING: If anything goes wrong at this step, very often this will make your whole computer unable to start and you may lose everything.
This step should be done by experienced users only.
Some tools to check are: CCleaner and TweakNow RegCleaner. (see http://xtechnotes.blogspot.com/2011/07/links-to-free-software.html )

5. Are all your files compressed?
Some computers may have come out of the box, configured to compress all your files by default. In Windows Explorer, if your files has filenames appearing in colour, then it may be compressed. To switch off this option, in Windows Explorer, right click on the folder and select Properties. Then uncheck any box for compression, for that file, folder or the entire drive.

6. Check updates
Ensure all updates including Windows Update and other antivirus updates are up to date.

7. Avoid re-installing Windows
Only reinstall Windows as a very last resort.

8. Cleaning Temporary Files
Ref: http://forum.wegotserved.com/index.php/tutorials/article/72-keeping-your-system-partition-cleaned-up-on-a-schedule/

C:\WINDOWS\system32\config\systemprofile\Local Settings\Temp\*.*
c:\temp\*.*
c:\windows\kb*.log
c:\windows\temp\*.*

- delete windows update uninstall files
dir c:\windows\$nt*

- delete Internet Explorer update uninstall files
dir c:\windows\ie8updates

..... any more ideas are appreciated ....

Sunday, January 01, 2012

Security Software Review, Discounts and Special Deals

This page lists some special offers on various security software including antivirus software and antispyware software. Since they are special deals from various sources, they are usually time limited. Please check the date when these links are posted.

There are also links to review of antivirus, antispyware, internet security software:

Android Security
A list of security software for Android devices is listed on this post:
http://xtechnotes.blogspot.com.au/2013/07/notes-android-apps.html

Reviews
The links in this section provide reviews of multiple antivirus, antispyware and security software. Sometimes the reviews contradict each other in their test results. However, looking at these reviews are better than not looking at all, when deciding which software to buy.
The Best Antivirus for 2012
Posted here: 2 Jan 2012
Review from pcmag
The main winners are: Norton Antivirus 2012 and Webroot SecurityAnywhere Antivirus

Reviews from CNET.com
Antivirus Software - review various software and can be filtered, sorted.

AV-Comparatives
Independent Tests of Anti-Virus Software

Virus Bulletion VB100
News on viruses and up-to-date review on antivirus software.

Lifetime Licences
The following antivirus, antispyware and security software have lifetime licences. They either let you pay once and use forever, or for multiple years. Unless otherwise stated, the list below are recommended software based on my positive experience.

WinPatrol Plus
"WinPatrol's easy tabbed interface allows you to explore deep inside your computer without having to be a computer expert. A one-time investment in WinPatrol PLUS provides a unique experience you won't find in any other software."

Spyshelter Premium
"SpyShelter uses special algorithms to protect you and your data against:
- Rootkits, zero-day malware, financial viruses that are used to steal or reveal your data to other parties and other harmful software for your system
- Extremely dangerous custom-made keyloggers and monitoring software that steal information you send via your computers- these are favorite tools of cybercriminals."

MalwareBytes Antimalware Pro
"Malwarebytes Anti-Malware PRO detects AND protects in an easy-to-use, straightforward, heavy-hitting but lightweight anti-malware application.
Consumers and personal users pay a one-time fee of just $24.95!"

Outpost Firewall Pro
http://www.agnitum.com/purchase/outpost/
"Outpost Firewall Pro provides a superior arsenal of defense against PC infiltration. Outpost ensures your online security with solid protection against all Internet-borne threats."

Discounts and Special Deals

Posted 14 May 2013
Bitdefender Internet Security 2013
https://partners.bitdefender.com/media/html/au/stgeorge/index.php
Appears to be intended for some Australian banks' customers. But the site allows anyone to download without checking any accounts. This may be a limited time.

Posted 24 Oct 2012
Malwarebytes deals at TrialPay is on again. This is a LIFETIME license product.
The catch is you do have to buy something from TrialPay, and the cheapest one I found is the Big Fish Games - about US$2.99.
The link below is the trialpay link specially for Malwarebytes - Enjoy!
http://www.trialpay.com/custom/cnet/downloadcom/?p=malwarebytes

Posted 12 Jan 2012

http://www.google.com.au/search?sourceid=chrome&ie=UTF-8&q=bit+defender+compatibility+malwarebytes
Bit Defender
FREE - This can be obtained for free by Westpac Bank or St George Bank customers in Australia, for 1 year I think. Very tempting because this seems like full software for free. But please look at the review for Bit Defender first (click on the link just above, or just google 'bit defender compatibility malwarebytes')
Several users have reported serious problems with this BitDefender.

Posted 7 Jan 2012

Spyware Terminator 2012
FREE - there is a round about way to get this "almost" free.
Download from the link above, and install it. This will be the basic version without F-Prot Antivirus.
After install - open the application and choose to Upgrade. Hopefully this will lead you to Trialpay.
In Trialpay - choose Snapfish and sign-up to print 20 photos for free. They will charge A$2.95 for handling fee.Then go back and receive installation code for full Spyware Terminator 2012 with F-Prot.
Summary: you get 20 free photo prints and Spyware Terminator 2012 with F-Prot. for Free but with handling fee of A$2.95

Posted 6 Jan 2012

Malwarebytes
coupon code – enter BM6-3S7-665 for 20% off or alternatively try code B3S-6Q1-H54 or Q65-TRJ-G7J to save 15% during checkout (you might want to check the comments for newer user-submitted coupons, some might not be working, depending on the actual website where you place the order)

Posted: 2 Jan 2012

http://www.webroot.com/En_US/sites/land-3product-25-40-50-percent-offer/
Webroot SecureAnywhere Complete
Up to 50% discount
Generally, this Webroot software has excellent and fantastic review but this particular review article below is quite the opposite.
http://www.expertreviews.co.uk/software/1288609/webroot-secureanywhere-complete

http://store.expertreviews.co.uk/p24688-avg_anti-virus_2012_1-pc
AVG Anti-Virus 2012 [1-PC]
includes 1-PC, 1-Year license
RRP: A$51.50
Save: A$45.55 (88%)
Our Price: A$5.95
Time Remaining: 30 days

http://download.cnet.com/Avira-Antivirus-Premium-2012/3000-2239_4-10625882.html?tag=mncol;2
Avira Antivirus Premium $19.99 1PC/1Yr

http://www.samssoftware.com.au/avira-products/home-products.html
Avira Antivirus Premium 2012 [Avira AntiVirus 2012] Discount (10% off)
CODE: Avira10
AUD$21.19.

Avira Internet Security 2012 Discount (10% off)
CODE: Avira20
AUD$43.28.

https://zemana.plimus.com/jsp/buynow.jsp?contractId=2207786
Click here to purchase Zemana AntiLogger for $10, using “Softpedia-NY-Promo” discount code
Zemana AntiLogger Full
RRP: A$36.00
Offer: $9.53

Codes valid on 22 Jan 2013:
specialoffer 50% off
smartpon-zal50 50% off

Discount Websites
The following sites often have coupons or discounts for software:
http://www.retailmenot.com
http://www.bitsdujour.com

Free Security Software

Best Free Intrusion Prevention and Detection Utility for Home Use (HIPS)

Outpost Security Suite (Firewall and Antivirus) FREE - does not work on Windows Server products
http://free.agnitum.com/
"Agnitum is the first security vendor to deliver a fully functional free version of an Internet security suite for Windows users. Outpost Security Suite FREE 7.1 builds on the acclaimed antivirus, firewall and proactive protection technologies. The free solution employs modern techniques to prevent infections, data corruption and PC intrusions."

ZoneAlarm Antivirus and Firewall Free
http://www.zonealarm.com/security/en-us/zonealarm-free-antivirus-firewall.htm
"The only all-in-one free security that seamlessly integrates award-winning Antivirus and ZoneAlarm Firewall together for maximum protection and performance."

Wednesday, December 07, 2011

Promotion: Hungry Jacks $2 Whopper

All that programming is making me Hungry!
--- for Australian readers only ----

Thursday, May 17, 2012

Tuesday, May 15, 2012

Tuesday, May 01, 2012

Sunday, April 22, 2012

Friday, April 20, 2012

Thursday, April 12, 2012

Saturday, March 31, 2012

Sunday, March 04, 2012

Tuesday, February 28, 2012

Sunday, February 19, 2012

Thursday, February 16, 2012

Thursday, February 09, 2012

Wednesday, February 08, 2012

Thursday, January 19, 2012

Sunday, January 01, 2012

Wednesday, December 07, 2011

Subscribe To

Translate

Followers

Search This Blog

Blog Archive

Support xTechNotes Blog

Keywords

Related Books

Tech Blog Links

Other Links

Total Pageviews