Notes Android Apps

There are several other blog post here at xtechnotes.blogspot.com about apps for Android but they have been scattered. The goal of this post is specifically on Android Apps and presented in their relevant categories. 

One obvious point is that there are thousands of apps out there. The list below deliberately contains only a few apps per category. The goal is to list the top apps only, so readers don't need to face with hundreds of apps for the same functionality, especially many of the available apps are questionable.

For reference, the other older sites are:
http://xtechnotes.blogspot.com.au/2010/12/notes-zenithink-zt-180-epad-android.html
http://xtechnotes.blogspot.com.au/2010/06/mobile-applications-for-apple-android.html

All apps listed below are for Android (may be available for other Operating System as well) and totally FREE.

NEWS
Feedly - aggregates websites and blogs.
Pulse - aggregates websites and blogs with offline cache support?
Flipboard - creates personalized magazine from specified topics.
Zite - learns your reading habits and compiles into a magazine format.

PHOTOS
Instagram - take photo and edit with many filters.
Snapseed - photo editor with gesture control.

UTILITIES
Onavo Extend - track data usage of apps And compress data download.
AutomateIt - setup triggers and actions to automate tasks on device.
Astro File Manager
ES File Explorer
File Manager Explorer
Inkredible - Note taking app that is not sensitive to palm accidentally resting on the tablet.
WiFi Analyser and Surveyor
Disk Usage - DiskUsage Shows How Your Android's Storage Space Is Used

VOICE / VIDEO / TEXTS
There are numerous apps that allow for voice calls, video calls and text messages. The list is available in:
http://ausfinance.blogspot.com.au/2013/06/mobile-apps-free-apps-for-call-text.html

SOCIAL
Whatsapp Messenger - free sms send from phone.
1stavailable - Medical booking program.

BLOGS
Blogger
Wordpress

PRODUCTIVITY
Evernote - note taking, image capture, OCR on server side.
Receipts (Proximiant) - scan receipts

LIFESTYLE
Runkeeper - measures pace, distance ad calories.
Kindle - ebook reader.
Zeebox Au - electronic program guide for free tv and foxtel. 
Jamie's Recipes - instructions for a variety of dishes

MEDIA PLAYERS
MX player
RealPlayer
RockPlayer

MUSIC - Radio
Pandora Radio - specify artist, song or composer and it Will create a radio station with relevan music for free.
TuneIn Radio - listen to radio from local and worldwide stations.
Stitcher Radio
Live365

TV
Aview - Free TV from BBC.


GAMES
Clash of Clans Free
Real Racing 3 
Dead Trigger

TRAVEL
XE Currency - conversion of world currency with historic rates, works offline.
TripAdvisor Hotels Flights - research and review hotels, restaurants, flights.
Kayak - comparison for flights, hotels and car rentals.

GOOGLE
Handouts - live chat and video calls.
Translate - translate between different languages.
Maps - street and satellite maps.

SECURITY
Avast Mobile Security
Avira Antivirus Security
BitDefender Mobile Security
Lookout
Trend Micro Mobile Security
Eset Mobile Security & Antivirus
Qihoo 360 Mobile Security
Norton Mobile Security
Zoner Antivirus Free
Wiper - app for doing various things like messaging, songs, but in a secure manner

Notes PostgreSQL

Installation Notes
--prefix=/usr/local/pgsql # default


1)  ./configure --prefix=/home/chee/usr/pgsql --with-perl --with-python --enable-odbc --enable-syslog

2) gmake

3) gmake check

4) gmake install

5) add path to LD_LIBRARY_PATH, PATH, MANPATH

6) create user "postgresq" -> adduser postgres

7) Install database directory:
   cd /home/chee/usr/pgsql/
   mkdir data
   chown postgres /home/chee/usr/pgsql/data    # change owner of "data" to "postgres"
   su - postgres                               # install db as user "postgres"
   ~/usr/pgsql/bin/initdb -D /home/chee/usr/pgsql/data   # installing db
   
8) Starting the database:

/home/chee/usr/pgsql/bin/postmaster -D /home/chee/usr/pgsql/data
or
    /home/chee/usr/pgsql/bin/pg_ctl -D /home/chee/usr/pgsql/data -l logfile start

     This will start the server in the foreground. To put the server in the
     background use something like

#### does not seem to work
     nohup /home/chee/usr/pgsql/bin/postmaster -D /home/chee/pgsql/data \
         </dev/null >>server.log 2>&1 </dev/null &

     To stop a server running in the background you can type

###     kill `cat /usr/local/pgsql/data/postmaster.pid`
     kill `cat /usr/local/pgsql/data/postmaster.pid`

     In order to allow TCP/IP connections (rather than only Unix domain
     socket ones) you need to pass the "-i" option to "postmaster".

  4. Create a database:

     createdb testdb

     Then enter

     psql testdb

     to connect to that database. At the prompt you can enter SQL commands
     and start experimenting.






     --with-includes=DIRECTORIES

          "DIRECTORIES" is a colon-separated list of directories that will
          be added to the list the compiler searches for header files. If
          you have optional packages (such as GNU Readline) installed in a
          non-standard location, you have to use this option and probably
          also the corresponding "--with-libraries" option.

          Example: --with-includes=/opt/gnu/include:/usr/sup/include.

     --with-libraries=DIRECTORIES

          "DIRECTORIES" is a colon-separated list of directories to search
          for libraries. You will probably have to use this option (and the
          corresponding "--with-includes" option) if you have packages
          installed in non-standard locations.

          Example: --with-libraries=/opt/gnu/lib:/usr/sup/lib.

     --with-pgport=NUMBER

          Set "NUMBER" as the default port number for server and clients.
          The default is 5432. The port can always be changed later on, but
          if you specify it here then both server and clients will have the
          same default compiled in, which can be very convenient. Usually
          the only good reason to select a non-default value is if you
          intend to run multiple PostgreSQL servers on the same machine.

     --with-CXX

          Build the C++ interface library.

     --with-perl

          Build the Perl interface module. The Perl interface will be
          installed at the usual place for Perl modules (typically under
          "/usr/lib/perl"), so you must have root access to perform the
          installation step (see step 4). You need to have Perl 5 installed
          to use this option.

     --with-python

          Build the Python interface module. You need to have root access to
          be able to install the Python module at its default place
          ("/usr/lib/pythonx.y"). To be able to use this option, you must
          have Python installed and your system needs to support shared
          libraries. If you instead want to build a new complete interpreter
          binary, you will have to do it manually.

     --with-tcl

          Builds components that require Tcl/Tk, which are libpgtcl,
          pgtclsh, pgtksh, PgAccess, and PL/Tcl. But see below about
          "--without-tk".

     --without-tk

          If you specify "--with-tcl" and this option, then programs that
          require Tk (pgtksh and PgAccess) will be excluded.

     --with-tclconfig=DIRECTORY, --with-tkconfig=DIRECTORY

          Tcl/Tk installs the files "tclConfig.sh" and "tkConfig.sh", which
          contain configuration information needed to build modules
          interfacing to Tcl or Tk. These files are normally found
          automatically at their well-known locations, but if you want to
          use a different version of Tcl or Tk you can specify the directory
          in which to find them.

     --enable-odbc

          Build the ODBC driver. By default, the driver will be independent
          of a driver manager. To work better with a driver manager already
          installed on your system, use one of the following options in
          addition to this one. More information can be found in the
          Programmer's Guide.

     --with-iodbc

          Build the ODBC driver for use with iODBC.

     --with-unixodbc

          Build the ODBC driver for use with unixODBC.

     --with-odbcinst=DIRECTORY

          Specifies the directory where the ODBC driver will expect its
          "odbcinst.ini" configuration file. The default is
          "/usr/local/pgsql/etc" or whatever you specified as
          "--sysconfdir". It should be arranged that the driver reads the
          same file as the driver manager.

          If either the option "--with-iodbc" or the option
          "--with-unixodbc" is used, this option will be ignored because in
          that case the driver manager handles the location of the
          configuration file.

     --with-java

          Build the JDBC driver and associated Java packages. This option
          requires Ant to be installed (as well as a JDK, of course). Refer
          to the JDBC driver documentation in the Programmer's Guide for
          more information.

     --with-krb4[=DIRECTORY], --with-krb5[=DIRECTORY]

          Build with support for Kerberos authentication. You can use either
          Kerberos version 4 or 5, but not both. The "DIRECTORY" argument
          specifies the root directory of the Kerberos installation;
          "/usr/athena" is assumed as default. If the relevant header files
          and libraries are not under a common parent directory, then you
          must use the "--with-includes" and "--with-libraries" options in
          addition to this option. If, on the other hand, the required files
          are in a location that is searched by default (e.g., "/usr/lib"),
          then you can leave off the argument.

          "configure" will check for the required header files and libraries
          to make sure that your Kerberos installation is sufficient before
          proceeding.

     --with-krb-srvnam=NAME

          The name of the Kerberos service principal. postgres is the
          default. There's probably no reason to change this.

     --with-openssl[=DIRECTORY]

          Build with support for SSL (encrypted) connections. This requires
          the OpenSSL package to be installed. The "DIRECTORY" argument
          specifies the root directory of the OpenSSL installation; the
          default is "/usr/local/ssl".

          "configure" will check for the required header files and libraries
          to make sure that your OpenSSL installation is sufficient before
          proceeding.

     --with-pam

          Build with PAM (Pluggable Authentication Modules) support.

     --enable-syslog

          Enables the PostgreSQL server to use the syslog logging facility.
          (Using this option does not mean that you must log with syslog or
          even that it will be done by default, it simply makes it possible
          to turn that option on at run time.)

     --enable-debug

          Compiles all programs and libraries with debugging symbols. This
          means that you can run the programs through a debugger to analyze
          problems. This enlarges the size of the installed executables
          considerably, and on non-GCC compilers it usually also disables
          compiler optimization, causing slowdowns. However, having the
          symbols available is extremely helpful for dealing with any
          problems that may arise. Currently, this option is recommended for
          production installations only if you use GCC. But you should
          always have it on if you are doing development work or running a
          beta version.

     --enable-cassert

          Enables assertion checks in the server, which test for many "can't
          happen" conditions. This is invaluable for code development
          purposes, but the tests slow things down a little. Also, having
          the tests turned on won't necessarily enhance the stability of
          your server! The assertion checks are not categorized for
          severity, and so what might be a relatively harmless bug will
          still lead to server restarts if it triggers an assertion failure.
          Currently, this option is not recommended for production use, but
          you should have it on for development work or when running a beta
          version.

     --enable-depend

          Enables automatic dependency tracking. With this option, the
          makefiles are set up so that all affected object files will be
          rebuilt when any header file is changed. This is useful if you are
          doing development work, but is just wasted overhead if you intend
          only to compile once and install. At present, this option will
          work only if you use GCC.

     If you prefer a C or C++ compiler different from the one "configure"
     picks then you can set the environment variables CC or CXX,
     respectively, to the program of your choice. Similarly, you can
     override the default compiler flags with the CFLAGS and CXXFLAGS
     variables. For example:

     env CC=/opt/bin/gcc CFLAGS='-O2 -pipe' ./configure


###############################################
Starting a session (assume database "testdb" already exist)
   =>psql testdb

Information:
   =>select current_user;
   =>select current_timestamp;

System:
- case INSENSITIVE
- use ";" to end statement
- prompt "=>" is first prompt, "->" are subsequent prompts, ended with ";"


PHP connection:
1. In data/postgresql.conf
      tcpip_socket=true

2. In data/pg_hba.conf:
      # TYPE     DATABASE    IP_ADDRESS     MASK               AUTH_TYPE  AUTH_ARGUMENT

local      all                                           trust
host       all         127.0.0.1      255.255.255.255    trust
host       all         129.94.176.241 255.255.255.255    trust

3. Run "postmaster -i ........"




User:
1. Creating user from Unix shell:   createuser demouser1
2. Creating user from within pgsql:
   a) start PostgreSQL by:  psql
   b) creating new user: CREATE USER demouser2;
3. Changing user permissions:
        test=> ALTER USER demouser2 CREATEDB;
        test=> CREATE GROUP demogroup WITH USER demouser1, demouser2;
        test=> CREATE TABLE grouptest (col INTEGER);
        test=> GRANT ALL on grouptest TO GROUP demogroup;
        test=> \connect test demouser2
        You are now connected to database test as user demouser2.
        test=> \q

Commands:
;                        to end sentence
\g                       (go) to end sentence
\p                       (print) to display buffer contents
\r                       (reset) to erase or reset buffer
\q                       (quit)to exit pgsql
\l                       (list) to list databases in the system
\d                       to list all Tables in the database
                         ORACLE: select * from cat
\d TABLE                 to list all attributes or colums of the TABLE
\connect <DB> <USER>     connect to database DB as user USER

\i FILE                  to read/run SQL script
\o FILE                  to print results to file called FILE
\o                       to switch output back to STDOUT
\t                       to switch off column titles when displaying query results
\z                       to see ownership of db objects


Data Types   PostgresQL    Oracle      Description
----------------------------------------------------------------------
char string  CHAR(n)       CHAR(n)     blank-padded string, fixed storage
             VARCHAR(n)    VARCHAR2(n) variable storage length
----------------------------------------------------------------------
number       INTEGER                   integer, +/-2billion range
             FLOAT                     float pt, 15-digit precision
             NUMERIC(p,d)  NUMBER(p,d) user-defined precision and decimal
----------------------------------------------------------------------
date/time    DATE          DATE          date
             TIME                        time
             TIMESTAMP                   date and time

DATE - use
"show datestyle" or
"SET DATESTYLE TO 'ISO'|'POSTGRES'|'SQL'|'US'|'NONEUROPEAN'|'EUROPEAN'|'GERMAN'"

***** PostgresQL has even more data types than listed here.




***************
Creating table
CREATE TABLE friend (
                     firstname CHAR(15),
                      lastname  CHAR(20)   );

Inserting Values
INSERT INTO friend VALUES (
        'Cindy',
        'Anderson'    );

Selecting Records
SELECT <attribute/column> FROM <relation/table>;
SELECT * FROM friend;
SELECT <attribute/column> FROM <relation/table> WHERE <attribute> <OP> <value>;
SELECT <attrib> FROM <relation> ORDER BY <attrib1, attrib2> DESC;
<OP> = {=, <, >, ........}
use "\t" in psql to omit the column titles.


Delete records
DELETE FROM <relation>;       !!! delete all rows
DELETE FROM <relation> WHERE <attribute> <OP> <value>;


Update
UPDATE <relation>  SET <attribute> = <value> WHERE <attribute> = <value>;


Destroying table
DROP TABLE <relation>


Input / Output data using COPY
NULL is displayed as  \N
COPY table TO 'file' USING DELIMITERS '|';
COPY table FROM 'file';
COPY table FROM stdin;
COPY table TO stdout;
COPY table TO 'file' WITH NULL AS '\';          NULL as blanks
COPY table FROM 'file' WITH NULL AS '\';        NULL as blanks
COPY table FROM 'file' WITH NULL AS '?';        NULL as '?'

Copying across network
- use stdin, stdout or pgsql's \copy command

Notes BigData

Notes BigData
===============

Definition
Web Intelligence and Big Data course
Why Big Data
Hadoop Ecosystem
MapReduce
Miscellaneous
Analysis


Definition
===========
Ref:  http://www.intel.com.au/content/www/au/en/big-data/unstructured-data-analytics-paper.html

- All history until 2003 - 5 exabytes
- 2003 to 2012 - 2.7 zettabytes
- data generated by more sources, devices, including video,
- data are UNSTRUCTURED, texts, dates, facts. Traditional Analytics are Structured Data (RDBMS).
- Analytics = Profit. Gartner survey - outperform competitors by 20% for those who use Big Data.
-



Web Intelligence and Big Data (WIBD) course
======================================
50 billion pages indexed by google.



More surprising events is better news.
- if event has prob p, then
    information = log_2 (p)  bits of information.

Mutual Information (MI) - between transmitted and received channels
- need to maximise MI
- eg mutual information between Ad$ and Sales
- eg adsense - given webpage, guess keywords.

IDF = inverse document frequency
- rare words make better keyword.
- IDF of Word = Log_2 (N / N_w)
  where N = total docs, N_w = number of word 'Word' in total docs.

TF = Term Frequency
- number of times the terms appear in that specific document.
- more frequent words (in that doc) make better keywords.
- TF = freq of w in doc d = n_w^d

TF-IDF = term freq x IDF = n_w^d x log_2 (N/N_w)
- words with high TF-IDF are good keywords.

Mutual Information between all pages and all words is prop to
   SUM_d  Sum_w  {  n_w^d x log_2 (N/N_w)  }

Mutual Information:  Input F -- Machine Learning -- Output B
Feature F, Behaviour B are independent.
Entropy H(F), H(B)
Mutual Information I(F,B) = SUM_f, SUM_b p(f,b) log { p(f,b) / p(f).p(b) }
Shannon: H(F) + H(B) - H(F,B)

WIBD - Naive Bayes
===================
Consider problem: P(BUY / r,f,g,c) where r,f,g,c are feature or keywords in web shopping.
Bayes Rule: P(B,R) = P(B/R).P(R) = P(R/B).P(B)
Naive Bayes assume r,f,g,c are INDEPENDENT
   - can derive Likelihood
      p(r/B)*p(c/B)*p(other features /B) ..... p(B)
      ---------------------------------------------  = L
      p(r/notB)*p(c/notB)*p(other features /notB) ..... p(notB)

      so if L > 1 we have a BUY, L < 1 then no Buy.



WIBD - Learn
==============
input X = x1,x2,...xn  (n-dimensional)
output y = y1,y2, ... ym
function f(X) = E[Y/X] expectation
              = y1*P(y1/X) + y2*P(y2/X)
Classification(video 5.2)
- eg X=size, head, noise, legs, Y={animal names}
- eg X= {like, lot}, {hate, waste}, {not enjoy}, Y = {positive, negative}
Clustering - unsupervised
- allow us to get classes from data. Need to choose right features.
- used when we DON'T know outputs relationship to start with.
- by Defn Clustering are regions MORE populated than random data
- add random data so that Po(X). So that r= P(X)/P0(X) is large means there is clustering
  then f(X)=E[Y/X]=r/(1+r) y=1 for real data, y=0 for added random uniform data.
- find things that do together to form a cluster. Eg Negative sentiment: hate, bad - but no one need to tell us they are Negative to start with.
- other means of clustering: k-means, LSH
Rules
- finding which features are related (correlated) to be each other. ie trying to cluster the features, instead of clustering the data.
- compare data which are independent features: Po(X) =  P(x1) * P(x2) * ... * P(xn)
  where x1 = chirping, x2 = 4 legged, etc xi={animal features}
  eg P(Chirping) = number of chirping / number of total Data
- Associative Rule Mining
  if there are features, A,B,C,D, want to infer some rule, eg A,B,C => (infer) D
  high support P(A,B,C,D) > s;   technique is to find P(A)>s, P(B)>s etc first
  high confidence P(D/A,B,C) > c
  high interestingness P(D/A,B,C) / P(D)  > i
- Recommendation of books - customers are features of books and vice versa.
  Use latent Models: matrix m x n = m x k TIMES k x n
                  eg people x books = people x genre TIMES genre x books
  NNMF - Non-negative

features: unemployment direction, interest rate direction, fraud

WIBD - Connect
===============
Logic Inference
if A then B    is SAME as ~A OR B  
Obama is president of USA:  isPresidentOf(Obama, USA) - predicates, variables
IF X is president of C  THEN X is leader of C:    IF  isPresidentOf(X,C) THEN isLeaderOf(X,C)
Query: If K then Q, consider that the query means ~K OR Q is TRUE,
       also same as K AND ~Q is FALSE.
       So proving K AND ~Q is FALSE, this means If K Then Q.

WIRD - Prediction
==============
Linear Least Squares Regression:
  x(i,j) with j features to predict, i-th data points with results yi for the i-th point
  Minimizing f(x) = E(y/X) is same as minimizing error = E(y-f(x))^2,  
  ... so let f(x) = xT.f   where f is the features vector of unknowns.
  Finding vector derivative and equate to zero ->  xT.x.f - xT.y = 0
  R^2 used to measure linear regression
Non - Linear correlation
  - Logisitic Regression - f(x) = 1 / (1 + exp(-f^T. x))
  - Support Vector Machines - Data may be high order correlated, eg parabolic correlation etc.
 Neural Networks
   - linear least squares
   - non-linear like logistic
   - feed-forward, multilayer
   - feed-back, like belief network
Which Prediction technique
FEATURES   TARGET  CORRELATION         TECHNIQUE
num        num     stable/linear       Linear Regression
cat        num                         Linear Regression, Neural Networks
num        num     unstable/nonlinear  Neural Networks
num        cat     stable/linear       Logistic Regression
num        cat     unstable/nonlinear  Support Vector Machines
cat        cat                         Support Vector Machines, Naive Bayes, Other Probabilistic Graphical Models



Why Big Data
==============
eg why Google(MapReduce), Yahoo(PIGS), Facebook(Hive) have to invent new stack
Challenges
1. Fault tolerance
2. Variety of Data Types, eg images, videos, music
3. Manage data volumes without archiving. Traditional need archives.
4. Parallelism was an add-on
Disadvantages
1. Could not scale
2. Not suited for compute-intensive deep analytics, eg in web-world
3. price-performance challenge. uses commodity hardware, open-source


Hadoop Ecosystem (See NotesHadoop)
===================================

MapReduce (See NotesHadoop)
=============


Miscellaneous
==============
About our speaker: Bio: Ross is Chief Data Scientist at Teradata and currently works with major clients throughout Australia and New Zealand to help them exploit the value of ‘big data’. He specialized in deployments involving non-relational, semi structured data and analyses such as path analysis, text analysis and social network analysis. Previously, Ross was deputy headmaster of John Colet School for 18 years before working as a SAS analyst, a business development manager at Minitab Statistical Software and founder and lead analyst at datamilk.com.

Ross Farrelly has a BSc (hons 1st class) in pure mathematics from Auckland University, a Masters in Applied Statistics from Macquarie University and a Masters of Applied Ethics from the Australian Catholic University.

Analysis
=========
path analysis
text analysis
social network analysis
natural language processing

Notes Hadoop

Notes Hadoop
=============

References
Famous Websites and their Big Data
Hadoop Ecosystem
MapReduce
Pig Latin
Hive
Twitter Case Study




References
===========
http://searchcloudcomputing.techtarget.com/definition/Hadoop
http://radar.oreilly.com/2011/01/what-is-hadoop.html
http://www.computerworld.com/s/article/9224180/What_s_the_big_deal_about_Hadoop_?taxonomyId=9&pageNumber=3


Famous Websites and their Big Data
===================================
Facebook - data analytics build around Hive
LinkedIN - infrastructure build around Hadoop


Hadoop Ecosystem (See NotesHadoop)
===================================
Big Data - 3Vs (Volume, Velocity, Variety)

Hadoop

Hadoop Streaming - enables user to write Map function and Reduce function, in any language they want. This middleware component will make these functions work under the Hadoop ecosystem.

Sqoop - JDBC-based Hadoop to DB data movement facility. Can transfer from RDBMS to HDFS. Can transfer from HDFS to RDBMS.
- Use Case - Archiving Old Data. Using Sqoop, data from RDBMS can be easily push to Hadoop clusters. Storing data in Hadoop instead of using Tape archives is more cost effective, provide fast access when needed, use one single technology for old and new data hence single know-how.

Hive - Enables users to use SQL to operate on Hadoop data. Hive contains only a subset of standard SQL. May also be used to perform SQL joins with tables from different DB systems, eg on table from MySQL with another table from DB2 or even Spreadsheets.

Pig - "Apache Pig is a high-level procedural language for querying large semi-structured data sets using Hadoop and the MapReduce Platform.
Pig simplifies the use of Hadoop by allowing SQL-like queries to a distributed dataset." "instead of writing a separate MapReduce application, you can write a single script in Pig Latin that is automatically parallelized and distributed across a cluster. "

Fuse - a middleware that allow users to access HDFS using standard file system commands (in Linux).

Flume-ng (next generation) - enable a load ready file to be prepared and then transferred to RDBMS using the RDBMS high speed loaders. The functionality is covered by Sqoop.

Oozie - chaing together multiple Hadoop jobs.

HBASE - high performance key-value store.

All Open Source. Most of the components are Java based, does not mean users need to program in Java.


MapReduce (See NotesHadoop)
=============
- Message passing, data parallel, pipelined work. Higher level compared to traditional Shared Memory or Distributed Message Passing paradigms.
- programmer need to specify only Mapper and Reducer. Message passing handled by the implementation itself.


Pig Latin
==========
http://www.ibm.com/developerworks/library/l-apachepigdataquery/#list1
http://pig.apache.org/docs/r0.7.0/tutorial.html#Pig+Tutorial+File


Hive
=====
Ref: [1] http://hive.apache.org/docs/r0.9.0/

What is Hive?
- Is a data warehouse infrastructure built on top of Hadoop.
- Provides tools to enable easy data ETL, (Extract, Transform, Load)
- put structures on the data, and the capability to querying and analysis of large data sets stored in Hadoop files.
- HiveQL easy for people familiar with SQL.
- Enable MapReduce framework to be able to plug in their custom mappers and reducers to perform more sophisticated analysis that may not be supported by the built-in capabilities of the language.
- Hive does not mandate read or written data be in the "Hive format"---there is no such thing. Hive works equally well on Thrift, control delimited, or your specialized data formats.

What Hive is NOT?
- Based on Hadoop, which is a batch processing system, Hive does not and cannot promise low latencies on queries. The paradigm here is strictly of submitting jobs and being notified when the jobs are completed as opposed to real-time queries. In contrast to the systems such as Oracle where analysis is run on a significantly smaller amount of data, but the analysis proceeds much more iteratively with the response times between iterations being less than a few minutes, Hive queries response times for even the smallest jobs can be of the order of several minutes. However for larger jobs (e.g., jobs processing terabytes of data) in general they may run into hours.

In summary, low latency performance is not the top-priority of Hive's design principles. What Hive values most are scalability (scale out with more machines added dynamically to the Hadoop cluster), extensibility (with MapReduce framework and UDF/UDAF/UDTF), fault-tolerance, and loose-coupling with its input formats."


Twitter Case Study
===================
Ref: "Large-Scale Machine Learning at Twitter"; Jimmy Lin and Alek Kolcz

Hadoop - at the core of the infrastructure.
Hadoop Distributed File System (HDFS) - data from other DBs, application logs, etc are written real time or batch processed into HDFS.
Pig - analytics done using Pig, which is a high-level dataflow language. It compiles the Pig script into physical plans and executed on Hadoop.

How to Check for Botnets

Botnets is a network of zombie computers.
A zombie computer is one which has been infected with Botnet type Malware that is able to control the computer and do the bidding of the Command centre.

This article aims to list the ways or tools to check if your computer has been infected and become part of a botnet.

Some articles on checking for botnet activity:
http://readwrite.com/2009/03/15/is_your_pc_part_of_a_botnet#awesm=~o9ydyMjFYimWMA
Is Your PC Bot-Infested? Here's How to Tell
     http://www.pcworld.com/article/170546/how_to_clean_bots.html
How to better protect your PC with botnet protection and avoid malware
     http://www.microsoft.com/en-gb/security/pc-security/botnet.aspx


Some tools to identify botnet:
BotHunter http://www.bothunter.net/
SRI International  www.sri.com - the creator of BotHunter
RUBotted (Beta) - from Trend Micro,
Windows Live OneCare safety scanner - online virus scan

Checkpoint- the makers of ZoneAlarm - seem to have a product which may not have been released yet.
"The Check Point Anti-Bot Software Blade detects bot-infected machines, prevents bot damages by blocking bot C&C communications, and is continually updated from ThreatCloud™, the first collaborative network to fight cybercrime."

Build Budget PC Less Than $300

June 2013

The date is important in regards to any quote. Recently I thought of upgrading an old PC used for some younger family members. The old PC was running an Athlon XP of early 2000s vintages. The PC itself has got a mixture of component.

The challenge is to build a PC for less that $300. As shown in the table below, this can be done. The prices are quoted from www.msy.com.au in Australian dollars. The actual components have prices listed in the Price column. The Alt column shows the prices of alternative PC components.

The key components that need upgrading are CPU, Motherboard, RAM, Hard Drive and Case. The cost limit is enough to include a CPU Fan. The main design is to have the budget Celeron CPU which has integrated ATI Radeon graphics. The motherboard is chosen to include some legacy PCI slots to use the existing wireless adapter card. The lack of grunt of the Celeron is made up by the whopping 8GB RAM. The old DVD writer can be reused. A modern PC Case would improve the image of this budget PC.

Alt	Price	Components	Model	Details	Link
	$62.00	Motherboard	AsRock B75 Pro3-M	2PCI, USB3, UEFI, Socket 1155, SATA3	http://www.asrock.com/mb/Intel/B75%20Pro3-M/?cat=Specifications
$60.00		Motherboard	Gigabyte GA-B75M-D3V	1PCI, USB3, UEFI, Socket 1155, SATA3	http://www.gigabyte.com/products/product-page.aspx?pid=4151#sp
	$44.00	CPU	Intel Celeron 1610	Ivy Bridge, 22nm, 2.6GHz, 55W, DualCore, 2MB L3, 64bit	http://ark.intel.com/products/71072/
	$67.00	RAM	8G Kit 1600 Patriot-S	DDR3, 2-sticks
	$69.00	Hard Drive	WD Green EZRX 1TB	3.5" SATA internal
$20.00		CPU Fan	Thermaltake Contact 16	2 copper pipes, 100W	http://www.thermaltake.com/products-model.aspx?id=C_00001805
$22.00		CPU Fan	CoolerMaster Hyper TX3 EVO	3 copper pipes	http://www.coolermaster.com/product/Detail/cooling/cpu-air-cooler/hyper-tx3-evo.html
	$15.00	CPU Fan	DeepCool Gammaxx200	2 copper pipes, 95W	http://www.deepcool-us.com/Product/GAMMAXX200/
	$43.00	Case and PSU	Generic	ATX
	$0.00	Wireless Adapter	Use existing PCI wireless adapter
	$0.00	DVD RW	Use existing
Total	$  300.00

Notes 3D Modelling for 3D Printers

3D printers is now commercially viable. There is so much one can do with 3D printers and so many ways to approach it. This is a collection of notes on the subject of 3D printing. This article is meant to be dynamic and incremented.

An excellent broad coverage of 3D printing is in:
"How to Own (and Love) a 3D printer"
http://www.techhive.com/article/2043202/how-to-own-and-love-a-3d-printer.html
"Why Microsoft’s 3D Printing Rocks"
http://makezine.com/2013/07/04/why-microsofts-3d-printing-rocks/


Main things for 3D printing:
Thingiverse - the first place to check out for 3D printing. Has many designs.

Makerbot - the company that produces the MakerBot 3D printer. The link here has more links to useful sites with regards to 3D printing.


File Format:
It seems the most widely or common format for 3D printer design is the STL format. Keep in mind when searching for 3D editors, check if they support the STL format.



3D Modelling editors:
https://tinkercad.com/ - online 3D editor

NetFab http://www.netfabb.com/basic.php

Meshlab - http://meshlab.sourceforge.net/  - open source editor

Google Sketchup Google's 3D designer that can be used for 3D objects and buildings. It supports the STL format via a plugin. For more details on acquiring this plugin, see:
http://sketchupdate.blogspot.com.au/2013/01/stls-for-3d-printing-in-and-out-of.html


5 May 2015
Below shows the picture of some current models of 3D printers.

Notes BitCoin

Bitcoin has been receiving a lot of news lately, especially due to the fact it shot up from $35 to over $200 in a few months, although it went back down to over $100 again soon. This is a technical article contain various tips for those interested in using or creating bitcoins.


The technical details of bitcoin were first outlined in a research paper by Satoshi Nakamoto. The coins debut online in 2009.


Newer alternatives to Bitcoin are:
Litecoins, Terracoins, RuCoins


Some places that handle Bitcoins
Bitpay
Mt.Gox - Bitcoin exchange

Some news on Bitcoins:
Cyber currency bursts mainstream
http://www.smh.com.au/business/world-business/cyber-currency-bursts-mainstream-20130411-2hni3.html

Zuckerberg's nemeses revealed as Bitcoin moguls
http://www.smh.com.au/money/investing/zuckerbergs-nemeses-revealed-as-bitcoin-moguls-20130412-2hp5h.html


Talks about the economy of Bitconis
http://www.dailyreckoning.com.au/bumps-on-the-bitcoin-road/2013/04/27/

How to earn or create Bitcoins

Mining Guide for Bitcoins
https://www.weusecoins.com/en/mining-guide

How to get started using your GPU to mine for Bitcoins on Windows
http://www.newslobster.com/random/how-to-get-started-using-your-gpu-to-mine-for-bitcoins-on-windows

python OpenCL bitcoin miner
https://bitcointalk.org/index.php?topic=1334.0

Easy Ubuntu python OpenCL mining setup
https://bitcointalk.org/index.php?topic=2636

Security - Protection against Botnets

Work in Progress......

This is a collection of notes on protection against botnets

BotHunter - read about this in one article. If anyone has experience with this, please share your opinions.

I'll leave the definitions of Botnet to other articles. Imagine being assimilated by the Bog in Star Trek - "Resistance is Futile - You Will Be Assimilated".

The best cure is prevention. To prevent your PC being a zombie in a botnet, here are a few tips, which are also the same tips from keeping safe from virus or malware.
- Use Firewall - make sure settings are appropriate
- Use AntiVirus, AntiMalware - although some suggest these are not effective against botnet.

How to Identify is you are a zombie in a Botnet.
- Use Firewall. Set the Firewall to the Most Restricted / Paranoid Setting. Switch off all of your user installed programs that run during startup. Do the same to services. Reboot the PC. Look at pop-up Firewall alerts and see what items are trying to make a network connection by itself.

Useful Sites:
http://electronics.howstuffworks.com/how-to-tech/how-to-fix-zombie-computer3.htm
This site has many good related sites.

http://www.pcworld.com/article/170546/how_to_clean_bots.html

http://security.stackexchange.com/questions/12446/how-do-i-know-if-my-computer-is-being-used-for-a-botnet-based-ddos-attack
Many good answers from various people. see darknet.

Google: how to tell if your pc is botnet

Notes Google Nexus 7

Just got a Nexus 7, 32GB Wifi, at OfficeWorks price match as a result of Bing Lee sale.
Anyway, just thought I'd start up some notes, starting with some links.

"23 Essential Tweaks to Perform on Your New Nexus 7 Android Tablet"
http://www.tested.com/tech/android/63025-23-essential-nexus-7-tweaks/


"Welcome to the Nexus: A Beginner’s Guide to the Google Nexus 7"
http://reviewlagoon.com/welcome-to-the-nexus-a-beginners-guide-to-the-google-nexus-7/

"Google's Official Nexus 7 site"
http://www.google.com.au/nexus/7/


Connecting by Wifi - very important to follow these steps if you want to maintain hiding your local wireless SSID but still want the Nexus 7 to remember it.
- Let your wifi router hide SSID, that is do not let it broadcast SSID
- Find out the MAC from the Nexus 7. Go to: Settings - About Tablet - Status - Wifi Mac address
- Go to the router administration and add the Nexus' MAC address.
- Go to the Nexus wireless setting and Add Network. Go To: Settings - Wifi - tap the "+"  symbol to add new network.
- The do a scan for wireless networks on the Nexus. If it cannot find your network, unhide the SSID on your router, then scan with the Nexus again.
- Once the Nexus is connected to the wireless network, go to the router administration and hide the SSID again. From now, the Nexus should be able to connect wirelessly, even with SSID hidden.

Security
1. Do not allow
Settings - Personal - Security - Unknown Sources - uncheck

How to Shrink, Compress, Backup DVDs

This post shows a number of ways and tools to accomplish the following:
Goal: To backup a DVD which is originally more than 4GB onto one standard single DVD about 4GB. The end-product needs to be the single DVD playable on most DVD players, not just playable on PCs.

When: This can happen when your Wedding DVD, your Graduation DVD or some of your Personal DVD happen to be in a DVD with all the contents totalling, say 6GB.

Here are the different parts to convert and backup your Personal DVD into a single playable DVD. The actual process of conversion and backup is described further down.

A. Open DVD and Save to ISO
Use DVD Decrypter 3.5.4.0.
Load the DVD into the drive.
Open the Disc from DVD Decrypter.
Choose the Source as the Drive
Choose the Destination as a file.
Run the program.

B. Open DVD and Save to Compressed  ISO
Use DVD Shrink 3.2
Load the DVD into the drive
Select whatever VOB files and other Files.
Run Backup

C. Open 6GB ISO file and Save to Compressed  ISO
Use DVD Shrink 3.2
Load the ISO file in DVD Shrink
Select whatever VOB files and other Files.
Run Backup

D. Open 6GB ISO file as a Drive
Use MagicDisc from Magic ISO.
Choose the 6GB iso file and MOUNT using MagicDisc.
Now a virtual DVD drive appears with the contents of the 6GB file.

E. Burn Compressed ISO 4GB to single DVD
Use ImgBurn as the DVD burner.
Choose the Source as the 4GB compresses ISO image.
Choose the Destination as the real drive with blank DVD.
Burn ISO to DVD.

The actual process can include the following steps.
Option 1:
   A  ->  C  ->  E
Option 2:
   B  ->  E
Option 3:
   A  ->  D  ->  B  -> E

What I found is that there are many ways to do this. But it appears that option 2, may be the most efficient. Option 2 uses DVD Shrink to read your original 6GB DVD and compress to 4GB standard DVD ISO file. Then ImgBurn is used to burn the image.

Extras:
- There is no need to consider audio or DVD menu since they will be automatically included.
- ISOBuster 1.5 is another tool that allow you to examine the contents of the ISO file.
- VirtualBox - within the virtual machine, there is the capability to mount an ISO file as a virtual DVD drive.

Notes on Windows

NotesWindows  - Notes on Windows
=============

This includes NotesWinXP which is now made obsolete.
The content here is related to any version of Windows and will be specifically denoted.


Contents
=========
Roaming Profile - Encryption
Useful Registry Keys
Stop accidentally moving files or folders
Windows Update does not work anymore
Windows CMD commands
Application Location
MS Windows XP backup
Change Location of My Pictures, My Documents
Change Location of Libraries in Windows 7
Password Revealed on Windows
Known Folders can be Hidden - Windows 7
Boot Manager - Windows XP, Windows 7
How to Repair a Corrupt Windows 7 Installation
Windows Recovery Console Operations
Windows 7 Administrator account
Windows 7 - Where are those Options
Windows 7 - Create a System Repair Disc
Microsoft Outlook
Windows 8 - Confirm Delete
Windows 7/8 Install - Creating Bootable USB
Windows 8 - Startup
Windows 8 - Startup Folder
Removing Rogueware
Hidden Windows Devices


Roaming Profile - Encryption - WinXP
=============================
Problem: When roaming profile cannot be saved - the reason could be due to some files are being encrypted.

Solution: Find and remove the encrypted file. Use the tool "cipher" provided by WinXP.

    cipher /S:directoryName
... to check encryption status of all subdirectories

Ref:
Roaming decrypt
http://www.microsoft.com/technet/prodtechnol/winxppro/reskit/c18621675.mspx

Search windows "roaming profile" decrypt
http://au.alpha.yahoo.com/search/web?p=windows+%22roaming+profile%22+decrypt
search http://au.alpha.yahoo.com/search/web?p=windows+%22roaming+profile%22+decrypt


 

Useful Registry Keys
================================
Append Path: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths
System Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment
User path:   HKEY_CURRENT_USER\Environment
Reserved Drives Letters: HKEY_LOCAL_MACHINE\SYSTEM\MOUNTEDDevices

Stop accidentally moving files or folders  - WinXP
===========================================
It is very easy to accidentally move folders in Windows Explorer, simply by careless movement of the mouse and accidentally releasing buttons. This problem is also very easy to occur with trackpads on laptops. There is a way of preventing this accidental motion by making Windows pop up a dialog box to ask for confirmation.

"Do You Want To Move Or Copy Files From This Zone?"

To enable this feature,
1. Click Start -> Control Panel -> Internet Options.
2. Click “Security” tab, mark the zone “Local intranet”, and click the “Sites” button.
3. Un-tick “Automatically detect intranet network”
4. Tick “Include all local (intranet) sites not listed in other zones”
   Tick “Include all sites that bypass the proxy server”
   Untick “Include all network paths (UNCs)”  

Windows Update does not work anymore  - WinXP
=====================================
For some reason, Windows files may corrupt such that the Windows Update will not work anymore. To solve this:
1. Open a Command Prompt Terminal: Start - Run, type: cmd.
2. To stop the current windows update service, in the Command Prompt, type:
    net stop wuauserv
3. Re-register the windows update dll for WinXP 32bit
    regsvr32 %windir%\system32\wups2.dll
Re-register the windows update dll for WinXP 64bit
    regsvr32 %windir%\syswow64\wups2.dll


Windows CMD commands
=====================
Ref: http://ss64.com/nt/

When in a CMD terminal, to launch another CMD terminal and execute a script (test.bat)
- start cmd /k test.bat    - new window remains open
- start cmd /c test.bat    - closes the new window when script finishes

Some commands related to Volume Shadow Copy Service (Vista)
vssadmin List Shadows
vssadmin List ShadowStorage


Application Location
=====================
WinXP:
Netscape Bookmarks (Netscape v 8.1)
D:\Program Files\Netscape\Netscape Browser\defaults\profile\bookmarks.html


MS Windows XP backup
======================
Below describes the process to take when backing up Windows. Some folder directory sizes given here may give an indication of their relative or typical sizes, but they will obviously vary in different systems.

- Login as each user and i) Empty Recycle Bin, ii) Clear all Cache, Cookies
- Login in as Administrator (so that can backup Registry and other System State)
- Goto Control Panel - Performance and Maintenance - "Free up space in your hard disk"
- c:\windows\system32 = 1GB
- c:\windows\ folders except system32 = 790MB
- c:\windows\system32\dllcache = 467MB
- c:\windows\$NTUninstall* = 245MB
- c:\windows\Driver Cache = 87MB
- c:\windows\fonts = 89MB
- c:\windows\help = 46MB
- c:\windows\ime = 86MB
- To backup Registry - go to the GUI windows Backup Utility and check the "System State"
- Do Normal and then Incremental backup.


Change Location of My Documents
=================================
WinXP:
Change My Documents.
- Right click My Documents - Properties - change the Target Location
- by default My Documents is in C:\Documents and Settings\<user>\My Documents

My Pictures, My Music.
- It appears that these folders are not changeable and are set under My Documents.
- When My Documents location is changed, then these folders should follow too.


Change Location of Libraries in Windows 7
===========================================
Libraries in Windows 7 can be assigned to various locations. Libraries will source the physial location of target directories and present the contents as if they all belong to the library. To change the location of Libraries in Windows 7:
- open Explorer
- in the left hand pane, right click on the library icons
- select Properties
- a list of location is presented, which can then be modified


Password Revealed on Windows
==============================
Windows 7:
- Passwords apparently can be revealed by System Information for Windows.
- the System Information (Start - All Programs - Accessories - System Tools - System Information) on Windows XP do not seem to reveal passwords.
- to remove the passwords in Windows 7, do:
    Control Panel - User Accounts and Family Safety - Windows Credential Manager.



Known Folders can be Hidden - Windows 7
==========================================
Windows default folders like Music, Pictures, Videos, Searches are part of the operating system and is not easy to remove. They may be hidden in a number of different ways.
1. Move them to another location by:
- Right click and select properties of the Known Folder
- Go to Location tab
- use the button Move to specify new location.
2. Merge them into Documents folder.
- Select the Known Folder to move.
- For the target to move to, select Documents.
3. Remove Known Folders
- Go to Start button and type, GPEDIT.MSC
- Under User Configuration, click Administrative Templates, double click Windows Components, then Windows Explorer.
- In the Disable Known Folders setting, double click and select Enabled.
- Click Show and enter names of any folders to be disabled.
- Go back to main Windows Explorer and delete any of the Known Folders, and it will be permanenet.

For a list of Known Folders, http://msdn.microsoft.com/en-us/library/bb882665.aspx


Boot Manager - Windows XP, Windows 7
======================================
Boot Loader for Windows XP: NTLDR
Sits on the Primary Partition of the Hard Disk
Configuration file: boot.ini
To edit, use Notepad

Boot Loader for Windows 7 / Vista : Windows Boot Manager (BOOTMGR)
Sits on the Primary Partition of the Hard Disk
Configuration file: BCD (Boot Configuration Data)
To edit, use BCDEDIT


How to Repair a Corrupt Windows 7 Installation
================================================
http://www.computerworld.com/s/article/9221540/How_to_Repair_a_Corrupt_Windows_7_Installation?taxonomyId=89&pageNumber=3


Windows Recovery Console Operations
====================================
Firstly to enter into recovery console, by booting from Windows CD/DVD. When 'Welcome to Setup' appears, then press 'R'. Here is a list of commands that may be helpful for repairing or trouble shooting in Recovery Console mode.

listsvc        - list the services running. Example display drivers agp440.sys is listed
disable agp440 - to disable a service that is running and listed under listsvc.


Windows 7 Administrator account
==================================
This section is specifically for Windows 7. In previous versions of Windows, there may appear the Administrator user on the login page. The Administrator account does not appear on Windows 7 by default.

By default, the Windows 7 installation usually
- allow you to create a new user, say FirstUser, during install and will set it up with Administrator password.
- there is a hidden Administrator user, but it is not active and has no password.

To manage the Administrator account, eg to activate it or give it a password, etc:
- click Start
- right click Computer - Manage
- Go to System Tools - Local User and Groups - User
- perform any task including Administrator account.

Alternative way to make Administrator appear on login screen.
- click Start and in the Search box, type CMD
- right click on the CMD icon that appears, choose Run As Administrator
- in the Command Prompt type:
  net user administrator /active:yes
      or
  net user administrator /active:no


Windows 7 - Where are those Options
=====================================
To enable Folder Options, which used to be a menu within Windows Explorer, we now need to do this for Windows 7:
- go to Contorl Panel - Appearance and Personalization - Folder Options.


Windows 7 - Create a System Repair Disc
========================================

The GUI way to create a Repair Disc is to Saerch for "Create a Repair Disc"

However, that can fail like this: once click create button and error came out
    original title: the parameter is incorrect ( 0x80070057)
    Unable to create a system repair disc 0x80070057

In that case, an alternative to create Repair Disc is:

i.           Boot the system by inserting Windows 7 installation disc.
ii.         When it gets to the main setup window, click on Repair from the lower left corner.
iii.       Select Command Prompt from the main Repair menu.
iv.        At the command prompt type in recdisc, then ENTER.
v.          You'll get the popup to choose which drive letter to create the Recovery disc to.
vi.        Have a clean DVD in the DVD drive and select start.
This should get your System repair disc created.



Microsoft Outlook
==================
Before you say this section should not be in the Windows notes, we argue that it should simply because MS love to tie in the products to the Windows OS so intimately that it affects other Windows related stuff, as you will see.


How to save, manage, archive Microsoft Outlook *.msg files?

- If your messages are in the Notes folder, or if you are looking for Calendar, Inbox or other items, DON'T LOOK FOR NOTES folder. Everything in Outlook inside Inbox, Notes, Calendar and more are DUMPED into a single *.pst file.
- - Where is the *.pst files?
- - - In Windows 7, it is default in:
- - - C:\Users\cyk\Documents\Outlook Files\Outlook.pst     (main outlook file)
- - - C:\Users\<user name>\AppData\Local\Microsoft\Outlook\*.pst   (other email accounts imported into Outlook)

- Tricky, you may be smart and Open Outlook, go to the left hand panel and find Inbox, Notes. Then right click on Notes, select properties. This will show the Notes folder in location called: "Outlook Data File". Don't spend hours looking for this, it is in fact the Outlook.pst mentioned above.

- So you cannot access all your messages without using Outlook, because it is also dumped into the Outlook.pst which is a proprietary format. Here's how to get all the messages out.
- - Open MS Outlook.
- - Open a Windows Explorer window and place it next to the Outlook app.
- - From the Outlook, click Notes, click the big buttons in the Ribbon to change to "Notes List". This give you a list of all the Notes in the main pane of Outlook.
- - Highlight all the notes you can to archive and save elsewhere. Drag them to a folder in the Windows Explorer window. Now all your notes will appear as individual *.msg files.
- - Download the little utility called "msgtxt" from:
http://www.enterag.ch/enterag/downloads/msgtext.xhtml
http://www.softpedia.com/get/Internet/Other-Internet-Related/MsgText.shtml
- - Use msgtxt to convert *.pst to *.txt files. A simple batch program can be written to simplify this.



Windows 8 - Confirm Delete
=============================
In Windows Explorer in Windows 8, it seems that it doesn't ask for confirmatin when you delete something.

Is this crazy or what? Not sure about those people who insists they don't make mistakes, but it is much safer to have the Confirm Delete option, which have been present in Windows many generations ago. So Windows 8 decide to trash this safety feature.

But you can bring it back - NOT from the Folder Options area.

You need to go to Windows 8 Desktop (no, not from the Metro UI), right click Recycle Bin.
Then click on Properties.
Then check the box 'Display delete confirmatin dialog'

If you want to test this, don't try deleting any important files ;)
Choose a dummy file that you don't mind losing and test this feature is back or not.


Windows 7/8 Install - Creating Bootable USB
=============================================
Creating the USB drive is the first step to installing Windows 7/8 from USB.

UEFI vs BIOS
For UEFI, the USB needs to be formatted as FAT32
For BIOS, the USB needs to be formatted as NTFS

When creating the USB on another PC, to check if that PC is booted with EUFI or not, launch "MSINFO32" on the CMD, then look at the BOOT MODE.

Size of USB drive needs to be at least 4GB


- Open a Command Prompt with Administrator privileges in Windows 7/8.
- Insert the USB device into an available USB port.
- Type "DiskPart" in the command prompt.
- Type "List Disk” (make note of the disk number of the target USB drive).
- Type "Select Disk X”, where X is the target USB drive noted in step above.
- Type "Clean”.
- Type "Create Partition Primary”.
- Select Partition 1
- Type "format FS=fat32 quick” for UEFI installation
    OR
  Type "format FS=ntfs quick” for BIOS installation
- Type "Active”.
- Type "Assign".
- Type "list volume".
- Type "Exit".


Extra step for UEFI
If the file 'BOOTX64.EFI' is not in \EFI\BOOT\ on the stick then copy the file to the stick by:
- go to a UEFI PC with Windows 7/8 x64, copy from C:\Windows\Boot\EFI for the file 'bootmgfw.efi to \EFI\BOOT\BOOTX64.EFI on the USB.


May NOT need to use Bootsec
- Mount the Windows ISO and access the ISO contents.
- cd to "boot" subdirectory
- type: bootsect.exe /nt60 X: where X is the drive letter for the USB key.
- this infuses boot manager code into USB key to make it bootable.


Ref: http://support.lenovo.com/en_AU/detail.page?DocID=HT076615
https://blogs.technet.com/b/home_is_where_i_lay_my_head/archive/2012/10/02/how-to-install-windows-7-or-8-from-usb-using-uefi.aspx?Redirected=true
http://www.thomas-krenn.com/en/wiki/Creating_Windows_UEFI_Boot-Stick_in_Windows
http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_install/problem-using-win7-usbdvd-download-tool-concerning/37eadf15-85dd-4e56-a89c-fa0a3339b92e
http://www.maximumpc.com/article/howtos/how_to_install_windows_7_beta_a_usb_key



Windows 7/8 Install - Installing from USB
=============================================

UEFI vs BIOS
With new Windowss and new Motherboards there is an option to use BIOS or UEFI.
Please consider this first.


1. Prepare a USB by first making it bootable, see - "Windows 7/8 Install - Creating Bootable USB"

2. Access the installation DVD for Win7/8.

3. Dump all the contents of the DVD onto the USB drive.
xcopy SRC_DRIVE:\* DST_DRIVE:\ /s /e

4. Use the USB to install on a new machine.


Windos 8 - Startup
===================
http://www.howtogeek.com/126016/three-ways-to-access-the-windows-8-boot-options-menu/

UEFI and Windows 8 Update on Windows/Linux dual-boot systems
http://www.zdnet.com/uefi-and-windows-8-update-on-windowslinux-dual-boot-systems-7000028217/

Seven ways to set up multi-booting with Windows 8 and Linux
http://www.zdnet.com/seven-ways-to-set-up-multi-booting-with-windows-8-and-linux-7000026392/

How to dual boot Windows 8 and Linux
http://apcmag.com/how-to-dual-boot-windows-8-and-linux.htm


Windows 8 - Startup Folder
=============================
Unlike previous versions of Windows, the Windows 8 startup folder is typically located at:
         C:\Users\<username>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Put any *.bat files here to make it run automatically at startup


Removing Rogueware
===================
- create restore point
- Use the following tools to scan and remove rogueware
AdwCleaner (www.bleepingcomputer.com/download/adwcleaner) to run scan
Junkware Removal Tool (www.bleepingcomputer.com/download/junkware-removal-tool) and run scan.
RogueKiller (www.bleepingcomputer.com/download/roguekiller) and run scan

Hidden Windows Devices
=======================
Some USB devices which have been unplugged or not active cannot be seen via Windows' Device Manager. However the following command will make Windows show the Hidden Devices. Open a CMD terminal, and enter the two lines:
- set devmgr_show_nopresent_devices=1
- start devmgmt.msc