xTechNotes - Technical Computing Programming Notes

Tuesday, October 30, 2007

Notes CSS

NotesCSS
=========

Contents
=========
References
Basic Syntax
Examples
Class Selectors
Inserting Stylesheet
Margins

References
===========
1. http://www.htmlhelp.com/reference/css/structure.html#pseudo
2. http://www.w3schools.com/css/css_syntax.asp

Basic Syntax
=============

General Syntax:
<Selector> { <Property> : <Value> ; [<Property> : <Value>] }

Selector is HTML tag, eg:
body, p, etc......
Property is HTML attribute eg:
color, font-family
Value is value of the HTML attributes

Examples
==========
body {color: black} -> simple example
p {font-family: "sans serif"} -> multiple worded Value
p {text-align:center;color:red} -> multiple Property

p -> alternative layout syntax
{
text-align: center;
color: black;
font-family: arial
}

h1,h2,h3,h4,h5,h6 -> multiple selector with same property
{
color: green
}

p.right {text-align: right} -> <p class="right"> This paragraph will be right-aligned.</p>

p.right {text-align: right} -> <p class="right centre"> This paragraph will be right-aligned.</p>
p.center {text-align: center}

.center {text-align: center} -> applies to ALL selector with class "center"

#green {color: green} -> applies to ALL id="green" selector, eg <p id="green"> .....

p#para1 -> applies to p selector with id="para1", eg <p id="para1"> .....
{
text-align: center;
color: red
}

input[type="text"] {background-color: blue} -> apply to selectors with certain elements, eg input-type

/* this is a comment */ -> comments

Class Selectors
================
<div class="sidenav">
<h2> Site navigation</h2>
</div>

div.sidenav { blah } /* styles overall div */
div.sidenav h2 { blah } /* styles h2 within the div */

ID Selectors
=============
#navigation { width: 12em; color: #333; }
div#navigation { width: 12em; color: #333; }

The major difference is that IDs can only be applied once per page,
while classes can be used as many times on a page as needed.

Classes can be used as many times as needed within a document.
IDs can only be applied once within a document.
So, if you need to use the same specific selector more than once, classes are a better choice.

Inserting Stylesheet
=======================

Cascading Order from least to most important:
- Browser default
- External style sheet
- Internal style sheet (inside the <head> tag)
- Inline style (inside an HTML element)

External style sheet
<head>
<link rel="stylesheet" type="text/css"
href="mystyle.css" />
</head>

Internal style sheet (inside the <head> tag)
<head>
<style type="text/css">
hr {color: sienna}
p {margin-left: 20px}
body {background-image: url("images/back40.gif")}
</style>
</head>

Inline style (inside an HTML element)
<p style="color: sienna; margin-left: 20px">
This is a paragraph
</p>

Multiple Sytlesheets
- in external CSS:
h3 {color: red;text-align: left;font-size: 8pt}
- in internal CSS:
h3 {text-align: right; font-size: 20pt}

Then the result will be:
color: red; text-align: right; font-size: 20pt

Margins
========

BODY { margin: 5em } /* all margins 5em */
P { margin: 2em 4em } /* top and bottom margins 2em,
left and right margins 4em */
DIV { margin: 1em 2em 3em 4em } /* top margin 1em,
right margin 2em,
bottom margin 3em,
left margin 4em */

Wednesday, October 24, 2007

Notes Windows Power Shell

Notes Windows Power Shell
=======================

Contents
=========
Commands
Scripting
Printing
Digital Signing and Certificates
Search

Commands
=========
get-alias list the real name of old command
get-content list contents of file
get-help get-service help on get-service command
get-command *-service list commands that manipulate services
get-member list the members of the object
get-service list the services running
| Out-Host -Paging prints one page at a time, user scrolling

Scripting
============
1. Extension *.ps1 (p-s-One)
2. To run script
/file.ps1 absolute path
./file.ps1 relative path
3. Line continuation '
4. Execution policy to run on local machine,
Set-ExecutionPolicy remoteSigned
5. Comments starts with #

Printing
==========
To print to a printer:
...blah... | Out-Printer -name "\\iaunsw024.au.cbainet.com\SYD48MP-L8-LXOP"

Search
=======
- Select-String
- findstr

Thursday, August 02, 2007

Notes Fortran

NotesFortran

Contents
=========

FORTRAN95 features
!DEC$ - compiler Directives
Keyword - SEQUENCE
Keyword - PURE
Initialize Data with slash /
Subroutines passed as argument of another subroutine
Allocatable Pointers / Array of Pointers
Using pointers in procedures
Dynamic Memory Allocation with Pointers
Declaration Statements for Arrays
Derived Data Types - with pointer components and being used as dummy pointer arguments
OpenMP
Static, Stack, Heap
Converting Integer to Character OR Writing to variable
Handling Character Strings between C# and Fortran
OpenMP programming warnings
Fortran Editors
String Manipulation

FORTRAN95 features
===================
List of F95 features implemented in Intel Fortran

1. FORALL -
2. PURE - for safety - ensure only INTENT(OUT,INOUT) arguments are changed.
3. ELEMENTAL - a type of PURE routine. Allow operation of arrays on element level.
4. CPU_TIME -
5. NULL intrinsic function - allow allocatable arrays to be pointed to this.

!DEC$ - compiler Directives
============================
Compaq (Digital) Visual Fortran (http://www.canaimasoft.com/f90vb/onlinemanuals/usermanual/TH_60.htm)

Compaq's Visual Fortran compiler (CVF) offers a great deal of flexibility to
create DLLs callable from C and/or Visual Basic. The compiler has a good set of
options to modify name-mangling, calling conventions and the method used to pass
arguments. Most of these options can be indicated through the use of compiler
directives (or pragmas) embedded in the source code. CVF compiler directives are
defined as comment lines, starting with DEC$.

To indicate that a subroutine should conform to the standard calling convention,
you add the DEC$ATTRIBUTE STDCALL compiler directive to the declaration of the
subroutine. For example:

subroutine MySub(argument1, argument2)
!DEC$ATTRIBUTES STDCALL:: GenDNASequence

In Compaq Visual Fortran, adding the STDCALL attribute also changes the default
method used to pass arguments, so you need to tell the compiler that arguments to
subroutine MySub are passed by reference. You can do this with the
DEC$ATTRIBUTE REFERENCE directive:

subroutine MySub(argument1, argument2)
!DEC$ATTRIBUTES STDCALL:: MySub
!DEC$ATTRIBUTES REFERENCE:: argument1, argument2

Also, when a procedure is declared with the standard calling convention (STDCALL),
Compaq Visual Fortran mangles its name. The name-mangling performed by CVF converts
the name of the procedure to all uppercase, adds an underscore as a prefix to the name,
and appends an at symbol (@) followed by the size of the stack (in bytes) at the end of
the name. The size of the stack is equal to 4 times the number of arguments in the
subroutine. MySub has 2 arguments, so the mangled name will be:

_MYSUB@8

Having to use this name to call our DLL procedure would be awful in most languages,
and illegal in Visual Basic. You can use another DEC$ATTRIBUTES compiler directive to
indicate an alias for the mangled name of the exported subroutine:

!DEC$ATTRIBUTES ALIAS: 'MySub'::MySub

The first argument is the alias (i.e. the name by which the subroutine would be available
to external programs using the DLL), the second argument is the Fortran name of the subroutine.

Finally, to indicate that the subroutine must be exported to the DLL as a public procedure,
you add the DEC$ATTRIBUTES DLLEXPORT compiler directive to the body of the subroutine:

!DEC$ATTRIBUTES DLLEXPORT:: MySub

So the full declaration of MySub would look like this:

subroutine MySub(argument1, argument2)
!DEC$ATTRIBUTES STDCALL:: MySub
!DEC$ATTRIBUTES DLLEXPORT:: MySub
!DEC$ATTRIBUTES ALIAS: 'MySub'::MySub
!DEC$ATTRIBUTES REFERENCE:: argument1, argument2
…

To create a DLL, you compile and link using the /dll switch. The following command-line
would compile and link MySub.f90 (containing subroutine MySub) into MySub.dll:

f90 MySub.f90 /dll /out:MySub.dll

Keyword - SEQUENCE
===================
SEQUENCE cause the components of the derived type to be stored in the same sequence they are
listed in the type definition. If SEQUENCE is specified, all derived types specified in component
definitions must be sequence types.

Keyword - PURE
===================
Pure Procedures
A pure procedure is a user-defined procedure that is specified by using the prefix PURE (or
ELEMENTAL) in a FUNCTION or SUBROUTINE statement. Pure procedures are a feature of
Fortran 95.
A pure procedure has no side effects. It has no effect on the state of the program, except for the
following:
• For functions: It returns a value.
• For subroutines: It modifies INTENT(OUT) and INTENT(INOUT) parameters.
The following intrinsic procedures are implicitly pure:
• All intrinsic functions
• The elemental intrinsic subroutine MVBITS
• The intrinsic subroutine MOVE_ALLOC
A statement function is pure only if all functions that it references are pure.

Initialize Data with slash /
==============================
Variables are not auto-initialized. To initialize a variable as you declare it,
put the initial value between two slashes. This kind of initialization is done once
when the unit is first loaded, and hence, it should not be used in a subprogram that
gets invoked repeatedly (use an assignment instead). For symbolic constants,
initialization is achieved via the parameter statement.

character title*20 / 'York' /
integer*2 count / 0 /
real*4 amount / 1.0 /
!---------------------------------------------------
! Note that title will have 20 characters even though
! we stored only 4 (they will be padded by blanks).

Subroutines passed as argument of another subroutine
======================================================
subroutine subA( subB )

The subroutine named subB needs to be declared EXTERNAL and hence not be in a module.
If a function needs to be in the same module then can be done as follows:

module test
contains
subroutine subA(subB)
external subB
.....
subroutine subC()
end module
subroutine subB()
use test
call subC()
end subroutine

Allocatable Pointers / Array of Pointers
=========================================
REF: Fortran 90/95 for Scientists and Engineers, Stephen J Chapman
1. It is illegal to have array pointers of native data types in Fortran:
REAL, DIMENSION(:), POINTER :: PTR
- the dimension attribute refers to the pointer's target, not of the pointer.
- the dimension must be deffered shape and the size is the size of the target, not
the pointer.
2. It is legal to have array pointers by using derived data types:
TYPE :: ptr
REAL, DIMENSION(:), POINTER :: P
END TYPE
TYPE(ptr), DIMENSION(3) :: P1

REF: Intel Fortran Language Reference
1. In contrast to allocatable arrays, a pointer can be allocated a new target even if it is currently associated with target. The previous association is broken and the pointer is then associated with the new target.
2. If the previous target was created by allocation, it becomes inaccessible unless it can still be referred to by other pointers that are currently associated with it.

Using pointers in procedures
===============================
Pointers may be used as dummy arguments in procedures and may be passed as actual arguments to procedures.
1. If a procedure has dummy arguments with either POINTER or TARGET attributes, then the procedure must have an explicit interface.
2. if a dummy argument is a pointer, then the actual argument passed to the procedure must be a pointer of the same type, kind and rank.
3. a pointer dummy argument must not have the intent attribute
4. a dummy argument cannot appear in an ELEMENTAL procedure in Fortran95.

Dynamic Memory Allocation with Pointers
========================================
REF: Fortran 90/95 for Scientists and Engineers, Stephen J Chapman

REAL, DIMENSION(:), POINTER :: ptr1
ALLOCATE (ptr1(1:10))

This statement creates an unnamed data object of the specified size and the pointer's type and sets the pointer to point to the object. Because the new data object is unnamed, it can only be accessed by using the pointer. After the statement is executed, the association status of the pointer becomes associated. If the pointer was associated with another data object before the ALLOCATE statement is executed, then that association is lost.

The data object created by using the pointer ALLOCATE statement is unnamed and so can only be accessed by the pointer. If all pointers to that memory are either nullified or reassociated, with other targets, then the data object is no longer accessible by the program. The object is still present in memory, but it is no longer possible to use it => MEMORY LEAK.

If a piece of allicated memory is deallocated, then all pointers to that memory should be nullified or reassigned. One of them is automatically nullified by the DEALLOCATE statement, and any others should be nullified in NULLIFY statements.

Declaration Statements for Arrays
===================================
SUBROUTINE SUB(N,C,D,Z)
REAL, DIMENSION(N,15) :: IARRY !explicit shape array
REAL, C(:), D(0:) !assumed shape array
REAL, POINTER :: B(:,:) !deferred shape array pointer
REAL, ALLOCATABLE, DIMENSION(:) :: K !deferred shape allocatable array
REAL :: Z(N,*) !assumed size array

Automatic Arrays - local array in a function, whose size is one of the arguments
Adjustable Arrays - array which is an argument, whose size is also one of the arguments.

To use arrays efficiently, see Optimizing Applications -> Programming Guidelines -> Using Arrays Efficiently.
When passing arrays as arguments, either the starting (base) address of the array or the address of an
array descriptor is passed:
When using explicit-shape (or assumed-size) arrays to receive an array,
the starting address of the array is passed.

When using deferred-shape or assumed-shape arrays to receive an array,
the address of the array descriptor is passed (the compiler creates the array descriptor).

Automatic vs Save
Automatic variables can reduce memory use because only the variables currently being used are allocated to memory.

By default, the compiler allocates local variables of non-recursive subprograms, except for
allocatable arrays, in the static storage area. The compiler may choose to allocate a variable in
temporary (stack or register) storage if it notices that the variable is always defined before
use. Appropriate use of the SAVE attribute can prevent compiler warnings if a variable is used
before it is defined.

Derived Data Types - with pointer components and being used as dummy pointer arguments
========================================================================================
Given a derived data type with pointer arguments:
type varDDT
type(BigDDT), pointer :: rComp
end type
Given that it is used as a dummy pointer argument
subroutine foo(aDDT)
type(varDDT), pointer :: aDDT
....
end subroutine

Then, in another function which uses "foo", the DDT can be used as:
program
type(varDDT), pointer :: aDDT_p
type(varDDT) :: vDDT

call testFoo(vDDT) ! vDDT contains the data found in testFoo
nullify(aDDT_p)
end program

subroutine testFoo(vDDT_local)
type(varDDT) :: vDDT_local
aDDT_p => vDDT_local

call foo(aDDT_p)
! vDDT_local will be able to pass to outside routine safely.
end subroutine

Note that this method is not required if the DDT concerned does not contain components
which are also DDT.

OpenMP
=======

Prerequisite:
Before inserting any OpenMP parallel directives, verify that your code is safe for parallel execution by doing the following:

Place local variables on the stack. This is the default behavior of the Intel Fortran Compiler when -openmp is used.

Use -automatic (or -auto_scalar) to make the locals automatic. This is the default behavior of the Intel Fortran Compiler when -openmp is used. Avoid using the -save option, which inhibits stack allocation of local variables. By default, automatic local scalar variables become shared across threads, so you may need to add synchronization code to ensure proper access by threads.

Static, Stack, Heap
====================
Heap area stores dynamic arrays
Static storage area store variables that are available for the life time of the program.
In C, local variables are stored in the stack.

In Fortran,
"By default, the compiler allocates local scalar variables on the stack. Other, non-allocatable variables of non-recursive subprograms are allocated in static storage by default. This default can be changed through compiler options. Appropriate use of the SAVE attribute may be required if your program assumes that local variables retain their definition across subprogram calls."

For openMP, local variables in Fortran will become automatic and thus stored in the stack. Note that
OpenMP threading model is based on threads and each has its own stack.

"static" as a descriptive term refers to the lifetime of C++ memory or storage locations. There are several types of storage:
- static
- dynamic (heap)
- auto (stack)
A typical storage layout scheme will have the following arrangement, from lowest to highest virtual memory address:
text (program code)
static (initialized and uninitialized data)
heap
(large virtual address space gap)
stack
with the heap and stack growing toward each other. The C++ draft standard does not mandate this arrangement, and this example is only an illustration of one way of doing it.

heap-array in Intel Fortran - This option puts automatic arrays and arrays created for temporary computations on the heap instead of the stack.
on Windows:
/heap-arrays- = no heap arrays (default)
/heap-arrays[:size] = where arrays of size (in kb) or larger are put on the heap.
on Linux:
-no-heap-arrays = no heap arrays (default)
-heap-arrays [size] = where arrays of size (in kb) or larger are put on the heap.
Example:
In Fortran, an automatic array gets it size from a run-time expression. For example:
RECURSIVE SUBROUTINE F( N )
INTEGER :: N
REAL :: X ( N ) ! an automatic array
REAL :: Y ( 1000 ) ! an explicit-shape local array on the stack Array X in the example above
! is affected by the heap-array option. Array Y is not.

Converting Integer to Character OR Writing to variable
========================================================
Example:
CHARACTER(LEN=15), ALLOCATABLE :: cPctile(:)
ALLOCATE( cPctile(nPctile) )
write(cPctile(jj), '(F15.7)') pctile(jj)

The number in pctile(jj) is being written into a CHARACTER variable called cPctile. Note the character has length of
15 which is enough to write 15 characters specified by the Format F15.7.

Handling Character Strings between C# and FortranSubmit New Article
Last Modified On : December 6, 2009 6:09 PM PST
Rate Please login to rate! Current Score: 0 out of 0 usersPlease login to rate! Current Score: 0 out of 0 usersPlease login to rate! Current Score: 0 out of 0 usersPlease login to rate! Current Score: 0 out of 0 usersPlease login to rate! Current Score: 0 out of 0 users

Handling Character Strings between C# and Fortran
=================================================================

Passing Character Strings as In parameters from C# to Fortran

C# provides built-in reference type "string" representing a string of Unicode characters. It is an alias for String in the .NET Framework. When passing "string" type by value from C# to Fortran function or subroutine platform invoke service copies string parameters, converting them from the .NET Framework format (Unicode) to the unmanaged format (ANSI), if needed. The unmanaged format is null-terminated so the C# method prototype of Fortran function or subroutine must account for the length argument passed along with the string address.

Passing Strings as In/Out parameters from C# to Fortran

Managed strings are immutable, platform invoke does not copy them back from unmanaged memory to managed memory when the function returns. If the Fortran function or subroutine wants In/Out parameters you need use StringBuilder Class in C#. StringBuilder Class represents a string-like object whose value is a mutable sequence of characters.

Fortran subroutine
subroutine FPassStringSub (Int_Arg, Str_In, Str_Out)
!DEC$ ATTRIBUTES DLLEXPORT :: FPassStringSub
integer, intent(in) :: Int_Arg
character*(*), intent(in) :: Str_In
character*(*), intent(out) :: Str_Out
end subroutine

C# method prototype
[DllImport("FDLL.dll", CharSet = CharSet.Ansi, CallingConvention = CallingConvention.Cdecl)]
public static extern void FPASSSTRINGSUB(ref int Int_Arg, string Str_In, StringBuilder Str_Out, int Str_In_Len, int Str_Out_Len);
STR_IN_LEN, int STR_OUT_LEN);

Returning Character Data Types from Fortran to C#

Similar to how C language handles Fortran function returning character data type the corresponding C# method prototype must add two additional arguments to the beginning of the argument list:
- The first argument is a StringBuilder object where the called function should store the result.
- The second argument is an int value showing the maximum number of characters that must be returned, padded with blank spaces if necessary.

Fortran function

function FRetString(Int_Arg)
!DEC$ ATTRIBUTES DLLEXPORT :: FRetString
character(*) :: FRetString
integer, intent(in) :: Int_Arg
end function

C# method prototype
[DllImport("FDLL.dll", CharSet = CharSet.Ansi, CallingConvention = CallingConvention.Cdecl)]
public static extern void FRETSTRING(StringBuilder Str_Result, int Res_Len, ref int Int_Arg);

How to avoid Race Conditions in OpenMP programming
===================================================
This section aims to cover a few points to watch for when doing parallel programming using OpenMP in Fortran.
The tips here could also apply to different languages with OpenMP such as C/C++.
1. First danger sign of parallel programming bug is inconsistent behaviour.
2. When a program stalls or hangs indefinitely sometimes, but other times run to completion.
3. When a program crashes sometimes, but other times run to completion.
4. When OpenMP is applied to a higher level function that calls many small low level functions, check whether module variables have SAVE attribute.
5. If module variables have save attributes, then check if those variables can be declared PRIVATE too so that no outside functions can use them.
6. When the SAVE module variables are being altered in a function in the module, check to see if the function can be encapsulated in a OMP parallel region.
7. When checking for race conditions using tools like Intel Thread Checker, ensure that the number of threads specified is not greater than the number of processors / cores.

Fortran Editors
=================

The following list are GUI based Fortran editors development environment.

Force
http://force.lepsch.com/

Plato - for Silverforst FTN95
http://www.silverfrost.com/16/ftn95/plato.aspx

Photran - on Eclipse IDE
http://www.eclipse.org/photran/

Geany - GTK based
http://www.geany.org/Main/AllFiletypes

There are many other general purpose editors that can edit Fortran including: Vi, Emacs, EditPlus.

String Manipulation
====================
There is a misconception that the string manipulating and handling capabilities are limited. However, it may not be as limited as initially thought. Here are a few useful string handling features from poplular languages that also exist in Fortran.

Comparing Substring:
INDEX( string, substring) -> returns integer representing the first position of occurance of the substring in string.
Converting Integer to String
write(charWord, '(I5)') ii-1
where ii is an integer which is written to the character variable 'charWord'.

Thursday, July 26, 2007

Notes Video Capture

Notes Video Capture
===================

See also NotesDVD

Contents
=========
Abstract
Capture Video via Virtual Dub
Experiment with Virtual Dub - Video Compression
Compress AVI (video and audio uncompressed) using Virtual DUB.
VirtualDub with DivX

VirtualDub filters plugins
VirtualDub MPEG2 codecs
How to Improve Video Capture Quality

Abstract
==========
These instructions uses
- Virtual Dub
- DivX codec
- mp3 codec
- video capture / tv tuner card

Video capture is done from miniDV camcorder into *.avi file using Virtual Dub.
The second stage compresses the *.avi file into another *.avi file.
Note between the two *.avi processes, an intermediate process is required to extract the sound from the first avi file into a separate wav file.

Capture Video via Virtual Dub
================================
1. Connect AV cable on Canon to COMPOSITE on TV card, yellow-video, red/white-audio

Do NOT
2. Switch on TV View program.
3. Select Video Source on TView program (select until display on PC screen)

DO:
2. Open Virtual Dub
3. File -> Capture AVI (to go to capture mode)
3.5 Video -> Video source -> Video Composite
4. Device -> Conexant Capture
5. File -> Set Capture File (give a filename)
6. Video -> Compression
Cinepak Codec = 6.6:1
Indeo Video 5.1 = 10:1
(Experiment with Virtual Dub - Video Compression)

Experiment with Virtual Dub - Video Compression
=================================================
Set the following:
- Capture - "Hidden display while capture", 29.97 fps
- Video - "Noise reduction enable", "Enable RGB filtering"
- Audio - MP3, 48kBit, 22kHz
- Record Duration about 20s

Uncompressed RGB YUV = 95.977MB
Cinepak Codec = 1.842MB
Indeo 5.1 = 2.734MB
Indeo 3.2 = 7.197MB
Indeo 4.5 = 2.591MB
Intel YUV = 55.044MB
MS Video1 = 41.639MB

MSVideo - Smoothest picture with smallest filesize

Audio better is using:
1) Raw (no compression)
2) Windows Recording Line Volume = 38%, better than higher volume.

Video - choose either
1) Intel YUV (better compression with DivX)
2) MS Video1

Compress AVI (video and audio uncompressed) using Virtual DUB.
===============================================================
1. Exit the capture mode from previous process
2. Open and select the *.avi file which contain the video and uncompressed audio from previous video capture process.
3. Audio - Source Audio, Full processing mode, Compression (mp3, 48kBit, 22kHz)
4. File -> "Save WAV ..." as MP3 - this process strips the sound from the previous *.avi file into a separate *.wav file.
5. Audio -> Audio From Other File - Select the file that was saved.
6. Video -> Full Processing Mode - Compression (DivX, High Quality)
for Good quality DivX, try
6.5 Audio -> Interleaving - adjust by testing if video out of sync with audio
7.File -> Save as AVI -> this will combine the video from the previous *avi and the newly saved *.wav file to produce a compressed DivX, MP3 *.avi file.
In terms of size, continuing from the experiment, the now compressed files have the following sizes:
IntelYUV_mp3 55.044MB -> 2.046MB
IntelYUV_raw audio 38%Vol 55.416MB -> 1.867MB
MSVideo1 raw audio 40.452MB -> 1.879MB
MSVideo1 raw audio 38%Vol 44.390MB -> 5.443MB

VirtualDub with DivX
======================
1. From the main Menu -> Video -> Compression -> DivX Codex -> Configure
2. In the DivX Codec Properties -> Main -> Profile = "High Definiion Profile" -> Rate Control = "1 Pass" -> Bitrate = 1500 kbps
Fairly good quality vs size -> Profile = "1080HD Profile" -> Encoding Presets = 8 -> Rate Control = "1 Pass" -> Bitrate = 3000 kbps

VirtualDub with DivX
======================
1. From the main Menu -> Video -> Compression -> DivX Codex -> Configure
2. In the DivX Codec Properties -> Main -> Profile = "High Definiion Profile" -> Rate Control = "1 Pass" -> Bitrate = 1500 kbps
Fairly good quality vs size -> Profile = "1080HD Profile" -> Encoding Presets = 8 -> Rate Control = "1 Pass" -> Bitrate = 3000 kbps

VirtualDub filters plugins
============================
Filter pack from Dee Mon:
http://www.infognition.com/VDFilterPack/
Jim Leonard's White Balance filter
http://neuron2.net/whitebalance/whitebalance.html

flaXen filter
http://neuron2.net/flaxen/flaxen.html

To use this filter, install Virtual Dub, then install these plugins into the VirtualDub's plugins folder.

VirtualDub MPEG2 codecs
========================
To use the Virtual Dub and encode with the MPEG2, the following codes need to be installed.
Panasonic VfW DV codec
http://www.free-codecs.com/download/panasonic_dv_codec.htm
Adaptec VfW DV codec
http://www.free-codecs.com/download/adaptec_dvsoft_codec.htm

How to Improve Video Capture Quality
======================================
This step may require additional filters for VirtualDub. See the previous section for filters available for VirtualDub.

White Balance Filter - Jim Leonard
- to correct for white balance problems.
- Example: when the video in general looks orange, blue or too dark.
- may occur when white balance is on automatic mode, so different types of light having different temperatures causes this problem
- this filter can also be used to adjust Hue, Saturation, Intensity, Brightness, Contrast

Deinterlacing filter
- used to remove the effect of interlacing, ie. when not all frames are processed.
- fast motion causes edges of objects to look jagged.
- the filter will also make the video look far sharper

Sharpening Filter
- used when video seem to have soft edges or lack detail.

Dynamic Noise Reduction
- used when video is grainy

Chroma Noise Reduction Filter
- used when there is chroma noise; ie where rainbow effects shimmer across the screen.

VHS filter - flaXen
- used when video has timing issues and skips a bit
- try using the Stabilize section of this filter only

NotesDVD

NotesDVD
=========
(see also NotesDVD)

Contents
=========
PAL/NTSC Aspect Ratio
DVD to DivX

PAL/NTSC Aspect Ratio
======================
Resolutions that video streams can use, are:

720x480 (NTSC, only with MPEG-2)
720x576 (PAL, only with MPEG-2)
704x480 (NTSC, only with MPEG-2)
704x576 (PAL, only with MPEG-2)
352x480 (NTSC, MPEG-2 & MPEG-1)
352x576 (PAL, MPEG-2 & MPEG-1)
352x240 (NTSC, MPEG-2 & MPEG-1)
352x288 (PAL, MPEG-2 & MPEG-1)

PAL/NTSC 720 x 576 / 720 x 480
Size Ratio
720 x 544 1.32:1
640 x 480 1.33:1
592 x 448 1.32:1
544 x 416 1.30:1
512 x 384 1.33:1
448 x 336 1.33:1
400 x 304 1.32:1
384 x 288 1.33:1
336 x 256 1:31:1
320 x 240 1.33:1

PAL/NTSC 720 x 576 / 720 x 480
Size Ratio
720 x 384 1.87:1
640 x 336 1.87:1
576 x 304 1.89:1
512 x 272 1.88:1
480 x 256 1.87:1
448 x 240 1.86:1

DVD to DivX
===========

Summary
1. Rip DVD VOB files from DVD to Hard Drive
2. Convert VOB into AVI
3. Convert VOB into WAV
4. Combine AVI and WAV into another AVI file

Method A: Using mpeg2avi, ac3decode, VirtualDub, Danii's GUI
Step 1 - assuming this is done .....
Step 2 - using mpeg2avi with Danii's GUI
Step 3 - using ac3decode with Danii's GUI
Step 4 - using VirtualDub

Convert VOB into AVI
- open Danii's GUI v0.20
- click MPEG2AVI
- fill in location of:
i) mpeg2avi program
ii) VOB file for single VOB file or
*.lst file for multiple VOB files (containing a list of all VOB files to be combined)
iii) output folder
- click on "DivX Auto"
i) Low motion
ii) 10 Keyframes
iii) 70% compression control
iv) 600 kbps (varies)
v) click Save
- fps = 25 (for PAL)
- q0 High quality
- r1 32bit MMX iDCT
- Output = o8 AVI-YV12 for DivX
- PAL - 4:3
- Crop and Resize (see Resolutions data above)
- Click Create My AVI

Convert VOB into WAV
- open Danii's GUI v0.20
- click AC3DEC
- fill in location of:
i) ac3dec program
ii) VOB file
iii) output folder
- Global Output Gain = 300
- Check - "Span over multiple VOBS automatically"
- Click Create My WAV

Combine AVI and WAV into another AVI file
- open Virtual Dub
- File -> Open video file (output from step 2)
- Video -> Direct Stream Copy (because already compressed to DivX/AVI)
- Audio -> WAV Audio
- Audio -> Full processing mode
- Audio -> Compression - MP3
- Audio -> Interleaving - adjust by testing if video out of sync with audio
- File -> Save As AVI

Saturday, July 21, 2007

Building own PC

Case:
Thermaltake Wing RS100
- $59 http://www.skycomp.com.au/ sydney

Windows Vista Readiness
- check out readiness from
http://www.microsoft.com/windows/products/windowsvista/buyorupgrade/upgradeadvisor.mspx

Thursday, June 14, 2007

NotesWeb

Registering your website
Resources

Resources
==========
https://fontawesome.com/?from=io
http://glyphicons.com/

Registering your website
============================
1. www.google.com/addurl
2. search.yahoo.com/info/submit.html
3. search.msn.com/docs/submit.aspx
4. www.dmoz.org

To analyse your website to see how spiders or bots rate your site:
Submit Express

Search Engine Optimization and SEO Tools

Web Hosting - free ones can be found in:
1. [free-webhosts](http://www.free-webhosts.com/)
2. [Best free web hosting of 2018](https://www.techradar.com/au/news/best-free-web-hosting-sites-of-2018)
- [Infinityfree](https://infinityfree.net/)
- [Freehostia](https://www.freehostia.com/)
- [5GBfree](https://www.5gbfree.com/hosting-plans/)
- [FreeHosting.com](https://www.freehosting.com/client/cart.php)
- [Byethost](https://byet.host/free-hosting)
- [x10hosting](https://x10hosting.com/free-web-hosting)

A few things to consider when choosing Free Webhosting.
1. When they say unlimited or infinite - that cannot be really true. Just remember this.
2. Check if they let you bring your own domain name, if you have already registered elsewhere.
3. If you would like to have multiple sub-websites, eg using multiple technologies like some using Joomla, some using Wordpress, etc, you may need to look for
i) how many databases do they offer?
ii) do they allow subdomains? For example your main site is www.main.com where you use Wordpress. Then you install Joomla on subA subdirectory, then the ability to make subA website to appears as subA.main.com is the subdomain.

For low-cost Webhosting sites - see:
https://www.hostingadvice.com/reviews/cheap/

Wednesday, May 16, 2007

NotesAudio

NotesAudio
===========

Filtering -> Remove Noise from Tape Recording

Filtering -> Remove Noise from Tape Recording
================================================
Using Goldwave Software.

Consider the following files:
SndOrig.wav -> original sound from tape recording with noise
SndNoise.wav -> Sample of white noise from SndOrig.wav
SngGood.wav -> the final improved sound file

1. Open Goldwave software and open the SndOrig.wav file.

2. Copy a section of from SndOrig.wav which is mostly noise (with no music)
and save to SndNoise.wav.

3. Copy the music part that we want from SndOrig.wav to SndGood.wav.
Select (highlight) and copy (Ctrl C) from waveform of SndOrig.wav.
Create a new file (File -> New) at 16bit, mono, 22050Hz (assuming original source is tape recording)
Paste (Ctrl V)

4. Open SndGood.wav and apply lowpass filtering by:
Effect -> Filter -> Lowpass/Highpass
Set initial cutoff freq to 2560Hz
Set to Lowpass
Set to Static
Click OK

5. On the SndGood.wav, increase volume by:
Effect -> Volume -> Change
Increase the volume to over 80% total

6. Open SndNoise.wav and "Select All" and "Copy". The noise wav is now in the clipboard.

7. Open SndGood.wav and reduce noise using the noise profile from clipboard by:
Effect -> Filter -> NoiseReduction
Select "Use Clipboard"
Click OK

Saturday, July 22, 2006

Notes Linux

Contents

- 32bit vs 64bit
- Bitmap Fonts
- CDwriting
- WinXP and Lilo booting sequence
- SSH Tunnelling Enabling and VNC
- SSH X11 port forwarding
- USB Mounting and /etc/fstab editing
- Loading Profiles
- Linux Firewall
- Linux Runlevels
- Block web-ads
- SELinux Configuration
- For using SELinux with cgi-bin and files in Symbolic links
- RPM
- Installing Source RPMs
- Linux Commands
- Others

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

32bit vs 64bit
==============

CPU 32 bit -> integer pointers are 32bits
2 to power 32 = 4e+09 about 4 Giga stuff

Real*8 is 8 bytes - i.e. 64 bits but these are a separate thing
altogether from integer pointers.

Pointers (a C/C++ concept - alien to Fortran)
Consider an array of REAL*8 Arr(17:27)
there are 11 elements, i.e. from Arr(17), ...., Arr(27)
each element can be referenced with a pointer so we need 11 pointers.
but if the pointer size is 2^32 (4e+09),
so the max array size(w.r.t. #elements) possible is 4e09

Bitmap Fonts
============
Eg. Installing Korean Fonts

0. Log in as root

1. Download Korean Fonts and put into /scratch/chee/fonts/Korean-X
The files are:
hanglg16.pcf.gz hanglm16.pcf.gz hanglm24.pcf.gz

If the files are in bdf format then,
i) convert to pcf:
bdftopcf .bdf
ii) compress it
gzip .pcf
or
compress .pcf

2. Create an index file for the fonts in that directory
mkfontdir /scratch/chee/fonts/Korean-X

this will create fonts.dir in the Korean-X directory

3. Tell the Xserver where to find the new fonts.

xset fp+ /scratch/chee/fonts/Korean-X
or
xset +fp /scratch/chee/fonts/Korean-X

to check the current X settings
xset q

to add this permanently, add the xset fp to .xinitrc

4. Refresh the Xserver setting when new fonts added if necessary

xset fp rehash

5. To view fonts in IRIX

xfd

CDwriting
=========
using cdrtools (cdrecord, mkisofs, etc..)
Ref: http://wt.xpilot.org/publications/linux/howtos/cd-writing/html/

1. Collect files and make a iso filesystem
mkisofs -r -o cd_image private_collection/

where
-r = Rockridge extns (deeper than 8 levels, permissions, etc)
-J = Joliet extensions - more MS Windows friendly
-o = output filename
-v = verbose
-x = exclude directory path
for second session only:
-C x,y = x is last session start, y is next session start
= use "cdrecord msinfo" to get -C x,y info
= must use with -M option
-M path = path where multi session is to merge. Usually path is
= place normally mount CDROM eg /mnt/cdrom
EG:
cd /scratch/chee/software
mkisofs -v -r -J -o firstImage.raw forWinPC/AdbeRdr601_enu_full.exe

*****
mkisofs 1.12.1 (i686-pc-linux-gnu)
61.14% done, estimate finish Fri Sep 10 16:32:27 2004
Total extents actually written = 8189
Total translation table size: 0
Total rockridge attributes bytes: 264
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used 5544
8189 extents written (15 Mb)
*****

2. Test Image
Mount the iso image (cd_image) into the directory /cdrom, so you can check the directories.

mount -t iso9660 -o ro,loop=/dev/loop0 cd_image /cdrom

3. Writing to CD
to test the CD writer and get info

cdrecord -scanbus

to write to the CD
shell> SCSI_BUS=0 # taken from listing 1 "scsibus0:"
shell> SCSI_ID=6 # taken from listing 1 "TOSHIBA XM-3401"
shell> SCSI_LUN=0
shell> cdrecord -v speed=2 dev=$SCSI_BUS,$SCSI_ID,$SCSI_LUN -data cd_image

# same as above, but shorter:
shell> cdrecord -v speed=2 dev=0,6,0 -multi -data cd_image

where
-dummy = simulation (laser off)
-eject = eject after burning
-v = verbose
-multi = multi-session
-msinfo = get multi-session info
-toc = get table of contents
-data = data file (mode1 YellowBook)
-audio = audio files
speed=x = x is speed for writing
dev = 0,0,0 for parallel.hpc.unsw.edu.au

EG
cdrecord -dummy -v speed=2 dev=0,0,0 -multi -data firstImage.raw

4. For CDRW
writing the 2nd, 3rd sessions onwards

shell> NEXT_TRACK=`cdrecord -msinfo dev=0,6,0`
shell> echo $NEXT_TRACK
shell> mkisofs -R -o cd_image2 -C $NEXT_TRACK -M /dev/scd5 private_collection/
EG.
cdrecord -msinfo dev=0,0,0
0,19591

mkisofs -v -r -J -o secondImage -C 0,19591 -M /dev/cdrom forWinPC/Firewall

cdrecord -dummy -v speed=2 dev=0,0,0 -multi -data secondImage

WinXP and Lilo booting sequence
===============================
- Example for Slackware Linux

- use the linux diskette to startup linux

- login as root

- modify the /etc/lilo.conf file as follow:

lba32
boot = /dev/hda1
vga = 773
image = /boot/vmlinuz
root = /dev/hda1
label = Slackware <-- no space read-only - then do: $ lilo -b /dev/hda1 <-- lilo on hda1 $ dd if=/dev/hda1 of=/dos/boot.lnx bs=512 count=1 $ shutdown -r now - take out the linux boot diskette and re-boot the system to WinXP, then do: ** copy the file from d:\dos\boot.lnx to c: ** then, modify the file c:\boot.ini to include following lines: [boot loader] timeout=30 [operating systems] c:\boot.lnx = "Slareware Linux 9.1" ... - re-start the Wxp and there should be an option on the Wxp loader to boot to Linux partition. - If you only want to boot to the Linux partition only, just set the active partition to the Linux installation partition (in this case, it is hda1) and reboot the Windows again. SSH Tunnelling and VNC ====================== ssh -L x:localhost:y anotherhost means connect to anotherhost using ssh and listen to port x on the local machine and forward any connections there to port y on anotherhost. To start a VNC server type: vncserver To start a VNC viewer type: vncviewer :

Using secure VNC with ssh, if a VNC server is running as display:1 on a system called linuxserver, you can start a secure SSH session by typing:

ssh -L 5902:localhost:5901 linuxserver

So now, connecting to port 5902 on a local machine actually connects to port 5901 on linuxserver. Thus to connect from a client machine:

vncviewer localhost:2

SSH X11 port forwarding
========================

On SSH server (remote host):
To check SSH is setup for X11 port forwarding, look at the file:
/etc/ssh/sshd_config
and see if the statement is present:
X11 Forwarding yes
then restart X11
/etc/init.d/sshd restart

On Cygwin (client):
- copy /etc/defaults/etc/ssh_config into ~/.ssh
- ensure the following are set:
ForwardX11 yes
ForwardX11Trusted yes
- DO NOT export DISPLAY=localhost:0.0

To ssh with X11 forwarding (may be different for different Unix/Linux)
ssh -X @

USB Mounting
=============
1. must be root to do this.
mount -t vfat /dev/sda1 /mnt/usb/

2. Alternatively, modify /etc/fstab file to have this line
/dev/sda1 /usbdisk vfat user,exec 1 1

Then any user can type
mount /usbdisk

and the files on the usb will have User's permissions

3. To unmount, type:
umount /mnt/usb/
or
umounut /usbdisk

Loading Profiles
================
/etc/profile -> /home//.profile -> /home//.bashrc

Master startup scripts are in /etc :
bashrc profile csh.cshrc csh.login

/etc/profile.d contains application specific Macro variables, eg.
java.sh java.csh

Linux Firewall
===============
/sbin/iptables --- executable
/etc/init.d/iptables --- script file
/etc/sysconfig/iptables --- configuration file

To stop firewall
/etc/init.d/iptables stop

To start firewall
/etc/init.d/iptables start

To check firewall
/sbin/iptables -L --line-numbers --- to see various rules
/sbin/iptables -nL --- to see various rules and ports

Chains
/sbin/iptables -F [chain] --- deleting all rules in the chain
/sbin/iptables -X [chain] --- delete optional user-def chain
/sbin/iptables -Z [chain] --- to reset / zero packet and byte counter
/sbin/iptables -N [chain] --- to create new user-def chain

: {Accept, Queue, Drop, Return}

Rule-specifications
/sbin/iptables -D chain rule --- delete rule in chain
/sbin/iptables -A chain rule --- append rule to end of chain
/sbin/iptables -I chain [rulenum] rule --- insert rule at rulenum or 1st position of chain
/sbin/iptables -R chain rulenum rule --- replace rule to at rulenum positin of chain

Rules specs: - < s/d > [!] IPaddr[/mask] [!] [port[:port]]
!=negate, []=optional

Other Parameters
-i --- interface, eg eth0
-p --- protocol, eg tcp, udp, icmp, all
-s --- source (see above)
-d --- destination (see above)
-j --- jump
-m state --- match with the state module
--line-numbers

Example - Webserver httpd - open at port 80
iptables -A INPUT -p tcp -i eth0 --dport 80 -j ACCEPT - Open HTTP port

or

So locate the rule in RH-Firewall-1-INPUT for WWW
eg iptables --line -L RH-Firewall-1-INPUT

eg 9 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:http
and insert a rule just after that rule (9 here) so insert before rule 10

# iptables -I RH-Firewall-1-INPUT 10 -p tcp -s 0/0 -d enrol1 -dport 8080 -m state --state NEW -j ACCEPT
If you are satisfied the rule is working save the ruleset with
/etc/init.d/iptables save
If you trash the ruleset but haven't saved just
/etc/init.d/iptables restart
(You will notice there is a preceding rule
7 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
which accepts incoming packets as part of an ongoing connection.)

To Save iptables:
/sbin/service iptables save

My Example for opening ports on local enrol1 to communicate with BIS on eb
iptables -I RH-Firewall-1-INPUT 12 -p tcp -s eb -d enrol1 --sport 49153:49155 -m state --state NEW -j ACCEPT

- To redirect all external traffic coming into port 80 into port 8080:
1. Use firewall configuration software to accept all TCP traffic from destination port 8080
2. (as root): iptables -t nat -A PREROUTING -p tcp --dport 80 -i eth0 -j REDIRECT --to-port 8080

- To redirect internal traffic coming into port 80 to port 8080
1. use program called rinetd.
2. in the /etc/rinetd.conf, have the line
"xx.xx.xx.xx 80 xx.xx.xx.xx 8080"
3. start the rinetd service

Linux Runlevels - Auto Startup
==============================
/usr/sbin/xxxd --- daemon or startup file to run
/etc/xxx.conf --- configuration file of the progran
/var/run/xxxd.pid --- process ID of the file that run

Runlevels
0 - system shutdown
1 - single user, no networking or daemons, for maintenance
2 - multiuser but no NFS, can access network but others cannot get in
3 - normal multiuser, no X login
4 - undefined
5 - multiuser X Display manager, X login from consoles
6 - reboot
s/S - single user mode

Program Sequence Config file
at Startup
===========================================
/sbin/init /etc/inittab

/etc/rc.d/init.d/* /etc/sysconfig/daemons

/etc/rc.d/rcN.d/*
(Sym links of init.d)

Note the files in rcN.d are preceded by K or S and a number
K = kill job
S = start job
number = order of pograms

To check what services are running:
/sbin/chkconfig --list

- Remember to check Linux Firewalls if network services do not start

- edit script files in /etc/init.d/ to contain
1. start/stop functions (and others functions restart)
2. commands in start function MUST NOT BE EXECUTABLE commands. They
should be executable scripts.
3. use daemon function to run as non-root user
4. DO NOT USE & (forking) at the end of command
5. Example:
daemon --user= 1> job.log 2>&1

- To register
/sbin/chkconfig --level 5 initdScript on

Block web-ads
==============

Most of us can't stand getting pummeled by ads everytime we go to a website. There
are a couple of ways to stop these ads. One is to edit your /etc/hosts file and list
your loopback address, 127.0.0.1, with the ad hosting address. For example, to
block ads from ar.atwola.com open up /etc/hosts as root:

[root@blueox]# vim /etc/hosts

Next, add the following line:

127.0.0.1 ar.atwola.com

This will line will tell your browser to look on 127.0.0.1 for the ads from
ar.atwola.com. Since localhost doesn't have the ad, it won't be displated.
You can download a hosts file that has a multitude of ads to block.
Personally I DO NOT RECOMMEND THIS approach. I have noticed that
my computer slows down trying to search through the hosts file.

The RECOMMENDED approach for firefox and mozilla users is to go to
the chrome directory of your personal browser folder.

[lumbrjackedpcj@blueox]# cd ~/.mozilla/firefox/dxshwykf.default/chrome

The directory dxshwykf.default will be different for you installation.

1. First, create the file userContent.css

[lumbrjackedpcj@blueox]# touch userContent.css

2. Next, edit this file with your favorite text editor

[lumbrjackedpcj@blueox]# vim userContent.css

3. Now, following the instrunctions on http://www.mozilla.org/support/firefox/adblock,
copy this userContent.css and paste it in yours.

SELinux Configuration
======================
Intro
The latest releases of Linux distributions incorporate the mandatory access
control architecture SELinux of the NSA into the Linux kernels.
More information on SELinux can be found at the following link:
http://www.nsa.gov/selinux/info/faq.cfm

/usr/sbin/sestatus -v --- to get status of selinux

En/Disabling SELinux:
1 - in your /etc/sysconfig/selinux file, set:
SELINUX=
... this is not TRUE disable, it only switches off enforcing and skips loading policy.

2 - if using GRUB as bootloader
go to /boot/grub/grub.conf or menu.lst (symlink to grub.conf)
add this line to the correct boot section
selinux=<0|1>

3 - if using LILO (... to be completed ..., contributions welcome)

Error Message:
"error while loading shared libraries: /usr/lib/liba52.so.0: cannot restore segment prot after reloc: Permission denied"
then your SELinux configuration is preventing your application from launching.

Solutions:
For shared libraries failing to load, try:
chcon -t texrel_shlib_t /usr/lib/liba52.so.0
For some other libraries
chcon -c -v -R -u system_u -r object_r -t lib_t $InstallationDirectory/SystemFiles/Libraries/

# View file/directory contexts: ls -Z /var/www
# Set file/directory context:
chcon -R -h -t httpd_sys_content_t /home/user1/public_html
Options:
* -R: Recursive. Files and directories in current directory and all subdirectories.
* -h: Affect symbolic links.
* -t: Specify type of security context.
# View security context of running processes: ps -eZ
# View security context of a user: id
uid=500(user1) gid=500(user1) groups=500(user1) context=user_u:system_r:unconfined_t
# Security contexts for a new file default to that of the parent directory.

For using SELinux with cgi-bin and files in Symbolic links
============================================================
Example:
perl script: /var/www/cgi-bin/overview
Data file: /home/BIS/BIS/log/ANGLE.txt
Symbolic Link: /var/www/html/BIS/ANGLE.txt

The security context, especially the subdirectories that contain it are
as follows:

/var/www/cgi-bin/overview root:object_r:httpd_sys_script_exec_t

/var/www/html/BIS user_u:object_r:httpd_sys_content_t
/var/www/html/BIS/ANGLE.txt user_u:object_r:httpd_sys_content_t

/home system_u:object_r:home_root_t
/home/BIS user_u:object_r:user_home_dir_t
/home/BIS/BIS user_u:object_r:httpd_sys_content_t
/home/BIS/BIS/log user_u:object_r:httpd_sys_content_t
/home/BIS/BIS/log/ANGLE.txt user_u:object_r:httpd_sys_content_t

Note that it is enough to ensure the "type" context at the /home/BIS/BIS
level and its contents seem to inherit the type.

RPM
===
Redhat package manager

Query
rpm -q -a --- list all packages
rpm -q -p --- list packages inside package_path rpm file
rpm -q -l -p --- list all files inside package_path rpm file
rpm -q -a -i --- list info on all packages
rpm -q -a -f --- list files of all packages

Installing Source RPMs
=======================
1. Get source code in rpm form from the "SRPMS" or "SOURCE" directories
in common repositories
http://download.fedora.redhat.com/pub/fedora/linux/core/5/source/SRPMS/

2. As root, unpack the source RPMs
rpm -ivh example.src.rpm

.... this will put the patches in /usr/src/redhat/SOURCES. But these
are not the real working file.
.... there is also a build *.spec file put in /usr/src/redhat/SPECS

3. Build the source code of the packages using:
command: rpmbuild -bp --target= kernel.spec
# cd /usr/src/redhat/SPECS/
# rpmbuild -bp --target=i686 kernel-2.6.spec
.... the source may appear in /usr/src/redhat/BUILD

4. Modify and recompile source
5. Package into rpm using:

Linux Commands
===============

XConfuration:
/usr/sbin/mouseconfig ------- mouse
chsh ------------------------- to change default shell
grep chee /etc/passwd -------- tells you what shell you are from

Xconfigurator ---------------- to use XFree86
/etc/X11/XF86Config ---------- configuration file
ls /usr/lib/X11/rgb.txt ------ to see a list of X11 defined color names

/usr/sbin/timeconfig --------- change time

/usr/sbin/ntsysv ------------- choose program to
/sbin/chkconfig -------------- boot during startup

/usr/doc/HOWTO --------------- HowTo help files

Commands:
useradd **** ----------------- add user with name ****
passwd ***** ----------------- make password for user
pwd -------------------------- print working directory
su **** ---------------------- to change to another user (from superuser)
su --------------------------- superuser
/sbin/shutdown -r now -------- shutdown and reboot now
startx ----------------------- to start X windows
ps -e/f/a/l/u ---------------- process
mount /dev/fd0 /tmp ---------- mount 1st floppy drive to /tmp
umount /tmp ------------------ umount any device from /tmp
cat notes >> notes2 ---------- prints the contents of a file to standard output
and then redirects and APPEND to file "notes2"
head [-count] ***** list first few lines of file
tail [-/+ [startline] [bcl]] *** list the last few lines
wc **** ---------------------- count the number of words, lines etc in a file
alias l='ls -al' ------------- make an alias, eg. to list files
ln **file1** **file2** ------- link file2 to original file 1 (hard link)
ln -s **file1** **file2**----- do symbolic link
ls -i ------------------------ list file and look and their i-node to see linked files
rsync -auv src dest ---------- copy / update between local/remote src/dest
size ------------------------- to calculate the size of the program
wc -l/c/w **file** ----------- Word Count lines, characters, words in a file
tr --------------------------- Translate, delete or replace text in a file.
tar cf - . | zip name - ***** Tar and then zip
unzip -p backup | tar xf - *** reverse of above - ie. unzip and untar
script ------------ starts a shell - records the whole sesion - exit the shell to stop script

Help Commands:
man -k **** ------------------ to search for keywords in manual entries
man # intro ------------------ #={1..8}, 8 intro topics of Unix manual.
man -M /opt/lam/man mpirun --- manual for "mpirun" program installed in /opt/lam
rpm -qdf /usr/bin/rtin ------- query
zless /usr/doc/HOWTO/...gz --- read compressed howto files without uncompressing
locate -d - find a command or file called filename
updatedb --localpath= --output=
to update data file before using locate
dirPath=path of file system to put in db
dbfile=database file to log info to.
slocate ...... replaces updatedb and locate commands
/etc/cron.daily/update ------- create database before using locate
find / -name .... ------------ find the file named ....
find ~ -size 0 -print -------- find file with zero size
crontab -e ------------------- to edit the crontab for current user

Build / Compile Programs
./configure --prefix=/home/che sets the installation to your local dir so no need root permissions

Environment:
set -------------------------- to list enviroment variables
xport AUTO=VW ---------------- set the variable auto=vw
alias ls='ls -color=auto' ---- colorize ls .. put this in .profile or .bashrc

./a.out ---------------------- to execute the file "a.out"
. ----------- execute command or script in CURRENT environment

Process Control:
kill -9 PID ----------------- sure kill the process
nohup **** ------------------ No hang-up for process ****
nice -# **** ---------------- reduce priority of command
nice --# **** --------------- increase priority of command
sleep **** ------------------ timer - stalls for certain time
at -l/r/m time date **** ---- timer - run command AT specified TIME

Compilers
cc -------------------------- SGI MIPS c compiler
CC -------------------------- SGI MIPS c++ compiler
gcc ------------------------- gnu c compiler
g++ ------------------------- gnu c++ compiler
insight --------------------- online help manual on SGI MIPS C/C++ compilers

Gnuplot
load "plotfile" ------------- runs the script file "plotfile"
set term postscript color
set output "file.ps" -------

Emacs

-------- change font size
.Xdefaults ------------------ change emacs properties
M-x replace-strings --------- to replace all occurances of strings
C-q C-m --------------------- produces the letters "^M"
M-x replace-strings, C-q C-m dos2unix file conversion within emacs
C-c C-w --------------------- Fortran End of Column marker
C-c C-r --------------------- Fortran ruler
M-C-j ----------------------- Fortran continuation line

Browsers:
netscape -install ----------- make colour look better

Graphics
gif2ppm --------------------- convert gif to portable pixmap format
pnmtogif -------------------- convert any pixmap to gif
pstoimg -transparent -------- make background transparent

Secure Operations
scp newfile chee@parallel.hpc.unsw.edu.au:~/programs

Networking
nslookup www.google.com ----- returns the IP address of the URL
ping ------------------------
ifconfig ---- configuration of adapter
netstat -in -----------------
echo 1 >/proc/sys/net/ipv4/ip_forward -- allow routing
echo 1 >/proc/sys/net/ipv4/conf/all/rp_filter -- prevent IP spoofing

System Admin / Setup
/etc/passwd ----------------- user details
/etc/sys_id ----------------- name of the system
/etc/hosts ------------------ list of IP address and Names
~/.rhosts ------------------- list of computers and usernames for rsh without password
background ------------------ customizing background of screen
ls -l /lib/libc-* ---------- check the glibc version
uname -r ------------------ check the kernel version
/etc/rc.d/rc.local ---------- startup programs are included here

System /Hardware Information
lsmod ----------------------- list loaded modules
lsdev ----------------------- display info about installed hardware
modinfo --------------------- information about installed modules
free ------------------------ display free/used memory on the system
more /proc/interrupts ------- view IRQ channels
more /proc/dma -------------- view dma channels
more /proc/ioports ---------- view I/O memory addresses

Change Resources
ulimit -a ------------------- check user limit
ulimit -aH ------------------ check user HARD-limit
ulimit -s 32678 ------------- set SOFT & HARD stack limit
ulimit -sS 32678 ------------ set SOFT stack limit
echo 8192 >/proc/sys/fs/file-max --- increase number of files openable
echo 16384 >/proc/sys/fs/inode-max --- increase number of files openable

File Commands
file ***** ------------------- prints the type of the file ****
find -c ---------------------- find file which has changed
stat -t ---------------------- provides info about file
diff file1 file2 ------------- output differences between files

Remote X Gui Interface
startX ----------------------- starts the X server
startkde --------------------- starts the KDE desktop

Modules
insmod ----------------------- insert module quick UNSAFE way
modprobe --------------------- insert module SAFE way

Others

dstat & sar ------------------ iostat, vmstat, ifstat and much more in one.
slurm ------------------------ Visualizes network interface traffic over time.
screen, dtach, tmux, byobu --- Keep your terminal sessions alive.
multitail -------------------- See your log files in separate windows.
tpp -------------------------- Presentation ("PowerPoint") tool for terminal.
xargs & parallel ------------- Executes tasks from input (even multithread).
duplicity & rsyncrypto ------- Encrypting backup tools.
nethack & slash'em ----------- Still the most complex game on the planet.
lftp ------------------------- Does FTPS. Can mirror, kinda like rsync.
ack -------------------------- A better grep for source code.
calcurse & remind + wyrd ----- Calendar systems.
newsbeuter & rsstail --------- Command line RSS readers.
powertop --------------------- Helps conserve power on Linux.
htop & iotop ----------------- Process, memory and io monitoring.
ttyrec & ipbt ---------------- Record and play back terminal sessions.
rsync ------------------------ Keeps filesystems in sync over SSH.
mtr -------------------------- traceroute 2.0.
socat & netpipes ------------- Directing stuff easily in and out of sockets.
iftop & iptraf --------------- To see where your traffic goes.
siege & tsung ---------------- Command line load test tools.
ledger ----------------------- Terminal-based accounting package.
taskwarrior ------------------ Todo management in the terminal.
curl ------------------------- Everybody's favorite HTTP toolbox.
rtorrent & aria2 ------------- Command line torrent downloaders.
ttytter & earthquake --------- Nice trendy Twitter clients :)
vifm & ranger ---------------- Alternatives to the midnight commander.
cowsay & sl ------------------ I just couldn't resist. :o)

Tuesday, January 31, 2006

NotesHTMLtags

from

HTML 4.01 / XHTML 1.0 Reference

NotesPython

Contents
=========
References
Style Guide
>Doxygen
Introduction
Compile
Tkinter - Python's GUI
Syntax
Data Types
Umask and file creation
Operators
Strings 
Lists
Collections - defaultdict
Math, Array, builtin functions
Flow Control
Functions, Modules, Append path
>Arguments
Global vars
Classes 
>Inheritance
>classmethod, staticmethod
Exceptions
>Passing Exceptions to outer functions
>Exception v2 vs v3
>Signals
Differences between Python 2.x vs Pyhon 3.x
>Exceptions
Importing 
File I/O
Writing CSV file
OS FileSystem and shell commands
>subprocess
Commands
Command line Options
Octo.py
Unix Script
Print
Check Text Format Newline CLRF
WARNINGS
<more complex things>
Sourcing Script within a python file
Python openedx
iPython /Jupyter
Symbolic Math Python
Printing Math Formula
Installing Packages
>Packages for Machine Learning
Zipfile and Datetime
PyCharm Editor
Python Logging module
Installing a Second Python
Python Package Manager - Setuptools and pip
Isolate Python Environment using virtualenv
@ Declarators
Python uploading to Gdrive
Python Writing Blog 
Troubleshooting
Miscellaneous
UnitTest
umask in python
Coverage
Codacy
Security - Bandit




References
===========
http://www.poromenos.org/tutorials/python
http://docs.python.org/tutorial/index.html
http://docs.python.org/py3k/tutorial/index.html
http://infohost.nmt.edu/tcc/help/pubs/tkinter/minimal-app.html - Tkinkter reference
http://www.tutorialspoint.com/python/python_gui_programming.htm
http://www.dreamsyssoft.com/python-scripting-tutorial/shell-tutorial.php

Library Ref: http://docs.python.org/2.7/library/
Language Ref: http://docs.python.org/2.7/reference/index.html

Style Guide
=============
https://www.python.org/dev/peps/pep-0008/



Example of docstring - using Triple quotes 

"""Example Google style docstrings.

This module demonstrates documentation as specified by the `Google Python 
Style Guide`_. Docstrings may extend over multiple lines. Sections are created
with a section header and a colon followed by a block of indented text.

Example:
    Examples can be given using either the ``Example`` or ``Examples``
    sections. Sections support any reStructuredText formatting, including
    literal blocks::

        $ python example_google.py

Section breaks are created by resuming unindented text. Section breaks
are also implicitly created anytime a new section starts.

Attributes:
    module_level_variable1 (int): Module level variables may be documented in
        either the ``Attributes`` section of the module docstring, or in an
        inline docstring immediately following the variable.

        Either form is acceptable, but the two should not be mixed. Choose
        one convention to document module level variables and be consistent
        with it.

.. _Google Python Style Guide:
   http://google.github.io/styleguide/pyguide.html
"""


>Doxygen
============
To use Doxygen, the python code must be like the following:
<code>
1 """@package docstring
2 Documentation for this module.
3 
4 More details.
5 """
6 
7 def func():
8  """Documentation for a function.
9 
10  More details.
11  """
12  pass
13 
14 class PyClass:
15  """Documentation for a class.
16 
17  More details.
18  """
19 
20  def __init__(self):
21  """The constructor."""
22  self._memVar = 0;
23 
24  def PyMethod(self):
25  """Documentation for a method."""
26  pass
</code>









Introduction
=============
Phython - What is it?
 - interpreted (no compilation)
 - interactive
 - object-oriented
 - compared with Tcl, Perl, Scheme, Java

Python - attributes
 - power and clear syntax
 - has modules, classes, exceptions, dynamic data types
 - interfaces to system calls, libraries,
 - interfaces to windowing systems X11, Motif, Tk, Mac, MFC

 - built-in modules can be written in C/C++
 - useable as an extension language for applications that need a programming interface.

Running Python
- run python script file *.py by:
   python.exe <filename>
... any file I/O assumes files are in the same directory.
- run python interactively using the GUI or Text console.
- run Python with Enthought Canopy editor
... any file I/O assumes files are in C:\users
... use os.getcwd() to find out current working dir
... use os.chdir(r'...') to change working directory
- run python without saying python:
eg file is pyVersion.py
--------------------
#/opt/pbs/default/python/bin/python
import sys
print (sys.version)
-------------------
To run this, just type:  ./pyVersion.py


Modules
- module file has python definitions and statements and called <modfile>.py
- to include this module to use in script or interactively:
    import <modfile>
- to use any functions, say foo, in <modfile>.py,
    <modfile>.foo
- to import specific functions only
    from <modfile> import foo
- to import everything
    from <modfile> import *

Run module file
    python <modfile>.py <arguments>

the module code has to include the following at the end 
----
if __name__ == "__main__":
    import sys
    fib(int(sys.argv[1]))
----
you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is executed as the “main” file:
So the example above will run the fib() function in <modfile>.py with arguments sys.argv[1] using the <arguments>


The Python interpreter reads a source file, and executes all of the code found in it. It will define a few special variables, eg __name__. If the python interpreter is running that module *.py as the main program, it sets __name__ to be "__main__". If this file is being imported from another module, __name__ will be set to the module's name.

The reasons for doing this is that sometimes you write a module (a .py file) where it can be executed directly. Additionally, it can also be imported and used in another module. By checking <if __name__ == "__main__":>, that code only execute when the module is run as a program and not run when the module is being imported into another code.


Run python script file (this is like module since it can contain function defs, but also have python executable statements)
 %run <filename>.py

B=A .....B is not a copy. It is a reference.


fileinput package:
----------
import fileinput
for line in file.input():
    print line

This will print out the lines of the files (eg data.txt) given to this python script. It can be run in TWO ways:
i) ./myprogram.py  data.txt
ii) ./myprogram.py < data.txt      (file indirection)




Compile
===========
$ python
>>> import py_compile
>>> py_compile.compile('server.py')
>>> ^D
$ python ./server.pyc




Tkinter - Python's GUI
======================

1. To use Tkinter, need
       from Tkinter import *

2. Set-up Root Window: 
 Tk()                           ### create Top-level window
 Tk.mainloop                    ### create Top-level window()

example:
  root = Tk()           ### create Top-level window
  myapp = MyApp(root)
  root.mainloop()  ### starts the EVENTLOOP to  listen to events
  root.destroy()        ### closes and exits this window called root


3. Containers:
 Frame(parent)                   ### starts a frame belonging to Parent
 Frame.pack()                    ### display the frame , geometry manager
 Frame.grid()                    ### geometry manager
 Frame.place()                   ### geometry manager



example:
   set up containers using:
       container1 = Frame(parent)

   ... where parent is the widget container1 belongs to.

   displaying containers, use:
       container1.pack()

   other geometry managers besides PACK, are GRID and PLACE


4. Buttons:
 Button(parent, attr=value,)        ### starts buttons and sets attributes
 Button.pack()                      ### displays the Button
 Button.configure(attr=value, ...)  ### sets some attributes
 Button["attr"]="value"             ### sets some attributes

   where attr = { text, background }


example:
   set up buttons:
       button1 = Button(parentContainer)

   display buttons:
       button1.pack()


5. Event Handling - i) Event Binding  ii) Command Binding
   Syntax:
 widget.bind( self, event, event_type_name, event_handler_name )

   event_type_name = { "<Button-1>", ....
 
   

Syntax
=========
Comments - # sign
Indentation - to specify blocks
Indentation - expected when statement end in colon ":" 
Line Continuation -  \



Data Types
===========
List, tuples, dictionaries, sets (sets library)
First (N items) array item index is 0, or -N
Last (N items) array item index is N-1 or -1

 List - 1D arrays, can also have List of Lists
Eg. list[:] - indexing single element
lstSample = [1, ["another", "list"], ("a", "tuple")]
lstList = ["List item 1", 2, 3.14]
lstList[0] = "List item 1 again"
lstList[-1] = 3.14
lstList[-2] =2
lstList[-3] = "List item 1 again"

Eg. list[i:j] - indexing / SLICING a sequence
think of the following sequence, where index are at the edge on in the middle.
0             1     2            3    -> Counter
  List Item 1    2    3.14000000      -> element sequence

>>> print lstList[:]          -> everything
['List item 1', 2, 3.1400000000000001]
>>> print lstList[0:2]        -> from right of index 0 to before index 2
['List item 1', 2]
>>> print lstList[-3:-1]      -> from right of index -3 to before index -1
['List item 1', 2]
>>> print lstList[1:]         -> from right of index 1 to the end
[2, 3.14]

Eg. list[i;j;k] - sequence with steps
>>> "hello"[::-1]             -> the whole sequence but with backward (-1) step
"olleh"


Eg. List Manipulation
lst1 = [1, 2, 3]
lst2 = [3, 4, 5]
>>> print [x * y for x in lst1 for y in lst2]
[3, 4, 5, 6, 8, 10, 9, 12, 15]
>>> print [x for x in lst1 if 4 > x > 1]
[2, 3]


 Tuples - immutable 1D array
Eg
tplTuple = (1, 2, 3)


 Dictionaries - hash tables or associative arrays
dicDictionary = {"Key 1": "Value 1", 2: 3, "pi": 3.14}
dicDictionary["pi"] = 3.15

Adding to dictionary:
default_data['item3'] = 3
default_data.update({'item4': 4, 'item5': 5})

Deleting item from dictionary
del dicDictionary["pi"]      # this will delete the "pi" key-value pair

dictionary can be intersected with list - see Lists section


Oct vs Int:
Python 2.6+
>>> oct(9)
'011'
>>> int(0o010)
8
>>> int(0o011)
9

Python 2.5-
>>> oct(9)
'011'
>>> int('010',8)
8
>>> int('011',8)
9


Umask and file creation
=============================
https://stackoverflow.com/questions/10291131/how-to-use-os-umask-in-python

os.stat
https://stackoverflow.com/questions/5337070/how-can-i-get-a-files-permission-mask
http://effbot.org/librarybook/stat.htm
 
https://stackoverflow.com/questions/5624359/write-file-with-specific-permissions-in-python

>>> import os
>>> os.getcwd()
'/home/chee/dev/Kyoto/tmp'
# Using default umask
>>> ff=open('pumask.txt','w')  
>>> ff.close()
>>> umask0002=os.umask(0066)
>>> print umask0002
2
# Using the 066 umask
>>> ff=open('pmask0066.txt','w')
>>> ff.close()
>>> umask0066=os.umask(umask0002)
>>> print umask0066
54
# using the reverted umask
>>> ff=open('pmask0002a.txt','w')
>>> ff.close()

-rw-rw-r--  1 chee chee       0 Dec 22 01:08 pumask.txt
-rw-------  1 chee chee       0 Dec 22 01:37 pmask0066.txt
-rw-rw-r--  1 chee chee       0 Dec 22 01:49 pmask0002a.txt


Or using os.open
------

import os
fd = os.open('/path/to/file', os.O_WRONLY, 0o600)
myFileObject = os.fdopen(fd)
myFileObject.write(...)
myFileObject.close()


Operators
==========
=  assignment
+=  add numbers or strings
-=  subtract
==  equality test
1 < a < 3 checks that a is both less than 3 and more than 1
del  delete|remove|clear|nullify variables, array elements or arrays


conditional operators:
==
!=
<>
>
<
>=
<=

Logical operators:
and
or
not = negate operator


 Variable Name assignment
Eg. 
Listt =[1,2]
fnVariable = len         # len is length function
print fnVariable(Listt)  # prints length of Listt



Strings 
=========
"blah" or 'blah' or "this 'is' blah" or 'this "is" blah'
''' this is 
multi-line string'''
"""this is 
multi-line string"""

unicode -> u"This is unicode string"


Find / Replace in Strings:
sout = "blah blah job_ID"
ii = sout.find("job_ID", beg=0, end=len(string))
-> returns index position of text found, or -1 if text not found.

sout.replace("$JOB_ID", jid_num)
-> replaces any "$JOB_ID" with string in jid_num


Value in Strings
eg. print "Name: %s\nNumber: %s\nString: %s" % (class.name, 3, 3 * "-")
Name: Poromenos
Number: 3
String: ---

Parse string:
In [8]: lhs, rhs = "2.7.0_bf4fda703454".split("_", 1)
In [9]: lhs
Out[9]: '2.7.0'
In [10]: rhs
Out[10]: 'bf4fda703454'


>>> "2.7.0_bf4fda703454".partition('_')
('2.7.0', '_', 'bf4fda703454')

>>> "shazam".partition("_")
('shazam', '', '')


"This is an example of \n\
   end of line string. But last one is \
   like that"

Substrings
   wordd = 'Hello World"
   wordd[0:2]        - 'Hel'
   wordd[:2]         - first two characters
   wordd[2:]         - from the first two to the last character.
   '<'+wordd*5+'>'   - concatenate with +, also repeat 5 times with *5.

   word = 'HelpA'
   word[-1]    'A'     - from the right end
   word[-2]    'p'
   word[-2:]   'pA'
   word[:-2]  'Hel'

Literal strings:   r'hello\world'   - treats the slash as a slash character, not something else.


Arguments in string:
sentence = "Hello {0}"
print sentence.format("Ted")    -> "Hello Ted"
print "Hello {0}".format("Ted")     -> "Hello Ted"
people="Ben"
place="Barn"
sentence = "I went .. {0} with {1}".format(people. placce)

String functions:
sentence.capitalize()
sentence.lower()
sentence.upper()
sentence.swapcase()
sentence.title()
sentence.count()
sentence.endswith()
sentence.isalnum()
sentence.isalpha()
sentence.isdigit()
sentence.islower()
sentence.isupper()
sentence.isspace()
sentence.lstrip()
sentence.rstrip()
sentence.strip()
sentence.replace()
sentence.split()
sentence.join()




Lists
======
a = ['spam', 'eggs', 100, 1234]
Indices start from 0, like strings
list.append()        - adds element to the end of list
list.pop()           - remove last element from list
list.extend()           - add last element to list

To have first in - first out, need deque:
    from collections import deque
the following is how to use the deque class.
myQ = deque([1,2,3])
myQ.append()         - adds element to the end of list
myQ.popleft()        - removes first element from the list.

mylist[-1]           - gives the last item on the list
mylist.index("blah") - returns index
"z" in my_list       - returns T/F to check if element is in a list
mylist[0:2]          - returns [0],[1] BUT NOT including [2] element
mylist.sort()
mylist.max()
mylist.min()
for item in mylist:  - looping over list elements

Prepend
a=['123','456']
a[:0] = ['abc']   This will Prepend 'abc' to a. Warning 'abc' MUST be in a list.

Intersection or Filtering both lists:
>>> adict = {"a": 1, "b":2, "c":3, "d":4, "e":5}    # define dictionary
>>> alist = ["a", "c", "z"]                         # define list
>>> [eee for eee in adict if eee in alist]          # results is the intersection of dictionary keys and list 
['a', 'c']
>>> for aaa in [eee for eee in adict if eee in alist]:
...    print aaa
...
a
c
#### results of looping over the intersection or list.

Flatten a list:
'\n'.join(myList)


Collections - defaultdict
===========================

>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k].append(v)
...
>>> d.items()
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]



Math, Array, builtin functions
================================


from the numpy library: import numpy
numpy.empty_like(a, dt, ...)    - make an empty array with same shape and size as a
numpy.ndarray                   - multidimensional homogenous array

Array(x:y) does not include y

mod and modulo (remainder) function
math.fmod(x,y)   -> x - n*y
a % b




Flow Control
==================
for .....:
 ...
elif:
   ....
else:
 ....

break      - break out of the for loop
pass       - do nothing
continue   - to skip remaining logic and go to the beginning next iteration


 
 For Loop using Range
lstRange = range(10)
for intNumber in lstRange:
    if intNumber in (3, 4, 7, 9):
        break
  elif intNumber == 5:
   continue
    else:
        pass # Do nothing
 
For loop using indices is specied using: range(1, n+1).
This will iterate from 1 until n. NOTE: the last integer is not iterated. Eg.
  for ii in range(1, numQuestions+1):

 If Loop
if lstRange[1] == 2:
    print "1 == 2"
elif lstRange[2] == 3:
    print "3 == 4"
else:
    print "Dunno"
 
 While Loop
while lstRange[1] == 1:
    pass


Functions, Modules, Append path
=====================================

 Normal Functions
# arg2 and arg3 are optional, they have default values
# if one is not passed (100 and "test", respectively).
def fnMyFunction(arg1, arg2 = 100, arg3 = "test"):
    return arg3, arg2, arg1
 
ret1, ret2, ret3 = fnMyFunction("Argument 1", arg3 = "Named argument")
 
- use "def" keyword to declare functions
- Optional arguments must have default values and listed after mandatory arguments
- Named argument is assigned a value
- can return a tuple of values


 Lambda Functions 
fnVariable = lambda x: x + 1
>>> print fnVariable(1)
2

a = ['spam', 'eggs', '100m', '1234']
b = ['m','s']
bFile = lambda x,y : x.endswith(y)
>>> bFile('spam', 'm')
True
>>> print map(lambda z: bFile('spam',z), b)
[True, False]
>>> print any(map(lambda z: bFile('spam',z), b))
True

Select elements of a list from another list
>>> rur_taskstats_keys = ["bkiowait", "etime", "minfault", "wcalls", "stime", "btime", "pgswapcnt", "abortinfo", "max_vm", "rcalls", "max_rss", "coremem", "majfault", "wchar", "utime", "exitcode", "rchar", "comm", "core", "ecode", "gid", "jid", "nice", "taskstats_nid", "pid", "pjid", "ppid", "prid", "rss", "sched", "uid","exitcode_signal"]
>>> rdef =["jid", "pid", "boof"]
>>> filter( (lambda x: x in rur_taskstats_keys), rdef)
['jid', 'pid']

Modules
- functions are placed *.py files to be able to be used elsewhere - these files are called modules.
- to use the functions defined in modules, say file is called func.py, then import using:
     import func
- python will search through the module search path.
- to determine the module search path, type:
    os.sys.path
- to append to the module search path:
     import sys,os,os.path
     sys.path.append(os.path.expanduser('~/Documents/My_Code'))
- NOT recommended to change PYTHONPATH.
- to automatically include the customised path, put the above sys.path.append(....) into a file like 00-start.py in this directory:
   ~user/ipython/profile_default/startup/00-start.py

>Arguments
============
THREE different forms of args are shown here:
def person(name, *args, **kwargs):
 print name
 for i in args:
  print i
 for key, val in kwargs.iteritems():      # each kwarg is a TUPLE (k,v), if no iteritems() then it will print just key
  print key, val

person("Jimmy", 1,2,3, gender="Male")
-- output--
Jimmy
1
2
3
gender Male
----------




Global vars
=============
To make use of global var, just need to declare that variable in all functions that use it, eg:
----------------------
globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print globvar  



Classes 
=========
Eg. 
class MyClass:   #OLD STYLE Python 2.x
class MyClass(object):   #NEW STYLE Python 2.x
#http://docs.python.org/release/2.2.3/whatsnew/sect-rellinks.html
    def __init__(self):
        self.varMyVariable = 3
    def fnMyFunction(self, arg1, arg2):
        return self.varMyVariable
 
    # This is the class instantiation
>>> clsInstance = MyClass()
>>> clsInstance.fnMyFunction(1, 2)
3

>Inheritance
==============
# This class inherits from MyClass. 
class OtherClass(MyClass):
    def __init__(self, arg1):
        self.varMyVariable = 3
        print arg1
 
>>> clsInstance = OtherClass("hello")
hello
>>> clsInstance.fnMyFunction(1, 2)
3
# This class doesn't have a .test member, but
# we can add one to the instance anyway. Note
# that this will only be a member of clsInstance.
>>> clsInstance.test = 10
>>> clsInstance.test
10


- Multiple inheritance is declared as:  class OtherClass(MyClass1, MyClass2, MyClassN)
- Private variables and methods names have double underscore "__"
- Dynamically add new member to class instance; eg. clsInstance.test


To call on the the parent's methods:
--------
class ChildB(Base):
    def __init__(self):
        super(ChildB, self).__init__()


classmethod, staticmethod
=============================
Ref: https://stackoverflow.com/questions/12179271/python-classmethod-and-staticmethod-for-beginner#12179752
https://johnsofteng.wordpress.com/2010/01/19/staticmethod-vs-classmethod-in-python/

Say function a() is defined in Parent Class, while Sub Class extends Parent Class

If function a() has @staticmethod decorator, Sub.a() still refers to definition inside Parent Class. Whereas,

If function a() has @classmethod decorator, Sub.a() will points definition inside Sub Class.

Let’s talk about some definitions here:

@staticmethod function is nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.

@classmethod function also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).

Example:
############
class Date(object):
    day = 0
    month = 0
    year = 0
    def __init__(self, day=0, month=0, year=0):
        self.day = day
        self.month = month
        self.year = year

@classmethod
    def from_string(cls, date_as_string):
        day, month, year = map(int, date_as_string.split('-'))
        date1 = cls(day, month, year)
        return date1
@staticmethod
    def is_date_valid(date_as_string):
        day, month, year = map(int, date_as_string.split('-'))
        return day <= 31 and month <= 12 and year <= 3999
############
    # usage:

day, month, year = map(int, string_date.split('-'))
date1 = Date(day, month, year)                # using basic way
date2 = Date.from_string('11-09-2012')        # using class method
is_date = Date.is_date_valid('11-09-2012')    # using static method



Exceptions
===========
def fnExcept():
    try:
        # Division by zero raises an exception
        10 / 0
    except ZeroDivisionError:
        print "Oops, invalid."


https://www.python.org/doc/essays/stdexceptions/
http://www.tutorialspoint.com/python/python_exceptions.htm
https://docs.python.org/2/library/errno.html


*** Principle is: if error is caught, then that is the end of the chain. 
To propagate the error, need to keep raising it in the exception.
---------------
import sys
def bye():
    sys.exit(40)                 # Crucial error: abort now!
try:
    bye()
except:
    print('got it')              # Oops--we ignored the exit
print('continuing...')

% python exiter.py
got it
continuing...
--------------

sys.exit(statuscode) is used to allow termination. This will raise a SystemExit exception. 

Do not use empty "except:" like above, because it will catch everything. Exceptions that are raised elsewhere will also be captured here. This may confuse the real cause of the problem.
The example above shows the system tries to exit, but that was caught by mistake and results in the program continuing.

To solve above problem, one can have:  
    except Exception:
the Exception is a superclass that does not include SystemExit so will not catch it. However other exceptions can still be caught unintentionally.

- these will let the program keep running
sys.exit(1)
sys._exit(1)
os.exit(1)
- this will stop the pogram immediately
os._exit(1)

sys.exit() is identical to "raise SystemExit()"
os._exit calls the C function _exit() and deos not do any clean up to flush buffers or clear handlers.

---


Defining Exceptions

class DeviceError(Exception):
 def __init__(self,errno,msg):
  self.args = (errno, msg)
  self.errno = errno
  self.errmsg = msg

# Raises an exception (multiple arguments)
raise DeviceError(1, 'Not Responding')



>Passing Exceptions to outer functions
=========================================

https://docs.python.org/2.7/tutorial/errors.html
>>> try:
...     raise NameError('HiThere')
... except NameError:
...     print 'An exception flew by!'
...     raise
...
An exception flew by!
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: HiThere


The last except clause may omit the exception name(s), to serve as a wildcard. Use this with extreme caution, since it is easy to mask a real programming error in this way! It can also be used to print an error message and then re-raise the exception (allowing a caller to handle the exception as well):

import sys
try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)
except ValueError:
    print "Could not convert data to an integer."
except:
    print "Unexpected error:", sys.exc_info()[0]
    raise


>Exception v2 vs v3
====================
see section "Differences between Python 2.x vs Pyhon 3.x" -> ">Exceptions"





>Signals
=========
https://docs.python.org/2/library/signal.html
import signal, os

def handler(signum, frame):
    print 'Signal handler called with signal', signum
    raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0)          # Disable the alarm


https://pymotw.com/2/signal/
$ python signal_getsignal.py
SIGHUP     ( 1): SIG_DFL
SIGINT     ( 2): &lt;built-in function default_int_handler&gt;
SIGQUIT    ( 3): SIG_DFL
SIGILL     ( 4): SIG_DFL
SIGTRAP    ( 5): SIG_DFL
SIGIOT     ( 6): SIG_DFL
SIGEMT     ( 7): SIG_DFL
SIGFPE     ( 8): SIG_DFL
SIGKILL    ( 9): None
SIGBUS     (10): SIG_DFL
SIGSEGV    (11): SIG_DFL
SIGSYS     (12): SIG_DFL
SIGPIPE    (13): SIG_IGN
SIGALRM    (14): &lt;function alarm_received at 0x7c3f0&gt;
SIGTERM    (15): SIG_DFL
SIGURG     (16): SIG_DFL
SIGSTOP    (17): None
SIGTSTP    (18): SIG_DFL
SIGCONT    (19): SIG_DFL
SIGCHLD    (20): SIG_DFL
SIGTTIN    (21): SIG_DFL
SIGTTOU    (22): SIG_DFL
SIGIO      (23): SIG_DFL
SIGXCPU    (24): SIG_DFL
SIGXFSZ    (25): SIG_IGN
SIGVTALRM  (26): SIG_DFL
SIGPROF    (27): SIG_DFL
SIGWINCH   (28): SIG_DFL
SIGINFO    (29): SIG_DFL
SIGUSR1    (30): SIG_DFL
SIGUSR2    (31): SIG_DFL


Differences between Python 2.x vs Pyhon 3.x
===============================================
http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html

https://docs.python.org/3/howto/pyporting.html

http://python-future.org/compatible_idioms.html


>Exceptions
====================
Python 2
print 'Python', python_version()
try:
    let_us_cause_a_NameError
except NameError, err:
    print err, '--> our error message'

result - Python 2.7.6
name 'let_us_cause_a_NameError' is not defined --> our error message


Python 3
print('Python', python_version())
try:
    let_us_cause_a_NameError
except NameError as err:
    print(err, '--> our error message')

result - Python 3.4.1
name 'let_us_cause_a_NameError' is not defined --> our error message





Importing 
===========
import random            # random library, to use do: random.<function name>
from time import clock   # time library, clock function
from numpy import *      # Dangerous since many libraries have some same function names
import matplotlib.pyplot as plt      # import and alias the function called pyplot

Random Numbers
=================
Python2.7
import random
random.seed()
random.randint(2, 99)   # generates random integers from 2 to 99


File I/O
=========
Eg. File I/O and serializing with "pickle" library

import pickle
lstList = ["This", "is", 4, 13327]
# Open the file C:\file.dat for writing. The letter r before the
# filename string is used to prevent backslash escaping.
flFile = file(r"C:\file.dat", "w")
pickle.dump(lstList, flFile)
flFile.close()
 
flFile = file(r"C:\file.txt", "w")
flFile.write("This is a sample string")
flFile.close()
 
flFile = file(r"C:\file.txt")
>>> print flFile.read()
'This is a sample string'
flFile.close()
 
# Open the file for reading.
flFile = file(r"C:\file.dat")
lstLoaded = pickle.load(flFile)
flFile.close()
>>> print lstLoaded
['This', 'is', 4, 13327]

# read all lines into a vector
flFile = open(r"tree_raw.txt", "r")
vlines = flFile.readlines()
oneLine = vlines[0].rstrip()    - removes the "\n" at the end of the word

# Functional Programming - to strip \n from words
manyWords = ['Applications\n', '-Browsers\n', '-Editors\n', 'Multimedia\n', '-Audio\n', '-Graphics\n', '-Video\n']
cleanWords = [line.rstrip() for line in manyWords]
result is ['Applications', '-Browsers', '-Editors', 'Multimedia', '-Audio', '-Graphics', '-Video']

# read contents of entire file
 f = open(file_name)
    try:
        return f.read()
    finally:
        f.close()

File modes: 
r rb r+ rb+      w wb w+ wb+          a ab a+ ab+ 
b      - for binary format
r,w,a  - for read, write, append
a      - if file exist, move pointer to end of file
w      - if file exist, overwrites
+      - for both reading and writing

Using 'with' keyword
- a construct that is similar to try-catch block, but used with unmanaged resources like file streams, and cleans-up even if there is exceptions
- example:
 with open("file.json") as json_file:
  # do some code - NOTE this is an INDENTED block

Appending a line to a file
 with open("_dollar.txt", "a") as outfile:
  outfile.write("$bin hello")

Opening files but checking first
------------------------------------
    if os.path.isfile(os.environ['PBS_CONF_FILE']):
        pbs_conf = open(os.environ['PBS_CONF_FILE'], 'r')
        for line in pbs_conf:
            os.environ[line.split("=")[0]] = line.split("=")[1].strip('\n')
        pbs_conf.close()
    else:
        print "Unable to find PBS_CONF_FILE ... " + os.environ['PBS_CONF_FILE']
        sys.exit(1)




Opening file with error catching:
-------------
try:
    f = open(my_file)

    try:
        do_stuff_that_fails()
    except EXPECTED_EXCEPTION_TYPES as e:
        do_stuff_when_it_doesnt_work()
    finally:
        f.close()

except (IOError, OSError) as e:
    do_other_stuff_when_it_we_have_file_IO_problems()
-------------
try:
    f = open('foo.txt')
except IOError:
    print('error')
else:
    with f:
        print f.readlines()




Open files using 'with':
-----------
with open(s_in, 'rw') as input_file:
 for line in input_file:
  count = count + 1
  #print str(count) + " " + line
  output = re.compile(r"^\s*[a-zA-Z0-9_]")   # searching line starting with non-whitesapce, non #
  result = output.search(line)    #result is a MatchObject or None
----------
try:
    with open(my_file) as f:
        do_stuff_that_fails()
except (IOError, OSError, Failure) as e:
    do_stuff_when_it_doesnt_work()
------------



Opening, Reading and Replacing lines in inline mode
--------
# Use the line number fond above, ADD a single line to the same input file.
count = 0
for line in fileinput.input(s_in, inplace=1):   #inplace=1 is having backup
 count = count+1
 if (count == lineNumberCount):
  print "WORKDIR"
 print line,
--------
this example prints line, but at the special line number, it prints and extra line with words "WORKDIR"


---from Alexis

import os
import stat

# Define file params
fname = '/tmp/myfile'
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL  # Refer to "man 2 open".
mode = stat.S_IRUSR | stat.S_IWUSR  # This is 0o600 in octal and 384 in decimal.

# For security, remove file with potentially elevated mode
try:
    os.remove(fname)
except OSError:
    pass

# Open file descriptor
umask_original = os.umask(0)
try:
    fdesc = os.open(fname, flags, mode)
finally:
    os.umask(umask_original)

# Open file handle and write to file
with os.fdopen(fdesc, 'w') as fout:
    fout.write('something\n')


Writing CSV file
===================
Ref:
http://stackoverflow.com/questions/2982023/writing-header-with-dictwriter-from-pythons-csv-module

from collections import OrderedDict
ordered_fieldnames = OrderedDict([('field1',None),('field2',None)])
with open(outfile,'wb') as fou:
    dw = csv.DictWriter(fou, delimiter='\t', fieldnames=ordered_fieldnames)
    dw.writeheader()
    # continue on to write data

---------- 

with open(infile,'rb') as fin:
    dr = csv.DictReader(fin, delimiter='\t')

# dr.fieldnames contains values from first row of `f`.
with open(outfile,'wb') as fou:
    dw = csv.DictWriter(fou, delimiter='\t', fieldnames=dr.fieldnames)
    headers = {} 
    for n in dw.fieldnames:
        headers[n] = n
    dw.writerow(headers)
    for row in dr:
        dw.writerow(row)



Ref: https://pymotw.com/2/csv/

import csv
import sys

f = open(sys.argv[1], 'wt')
try:
    writer = csv.writer(f)
    writer.writerow( ('Title 1', 'Title 2', 'Title 3') )
    for i in range(10):
        writer.writerow( (i+1, chr(ord('a') + i), '08/%02d/07' % (i+1)) )
finally:
    f.close()

print open(sys.argv[1], 'rt').read()


JSON in Python
=================
 import json
 with open(file_poly) as json_file:
        json_data = json.load(json_file)
        json_file.close()

        if (iDebug==1):
            print(json_data)
            a_min = json_data["a"]["min"]
            a_max = json_data["a"]["max"]
            print a_min, a_max
    return json_data

The above code is for the JSON file below:
-----------------
{
    "Number_of_Questions" : 10,
    "a":{
        "min":"0",
        "max":"10"
    },
    "b":{
        "min":"1",
        "max":"9"
    },
    "c":{
        "min":"2",
        "max":"8"
    }
}
-----------------


(#https://docs.python.org/2/library/json.html)
To construct a JSON data structure from strings:
-----------
import json
text = \
'{ \
    "kind": "blogger#post", \
    "blog": { \
        "id": "8070105920543249955" \
    }, \
    "title": "A new post", \
    "content": "With <b>exciting</b> content..." \
}'
parsed = json.loads(text)
print json.dumps(parsed , indent=4, separators=(',', ': '))
-----------
NOTE: important to have the line continuation





OS FileSystem and shell commands
=====================================
import os           - package that enables shell commands.
os.getcwd()         - print current working directory
os.chdir('path')    - change directory
os.listdir()        - list contents of directory
os.path.basename(v) - return the base name of pathname of v
os.environ['HOME']  - get the value for env var HOME
os.environ.get(var) - get value for environment variable 'var'
os.path.isfile      - check if file exist
os.path.isdir      - check if directory exist
os.sys.path         - list the search path
os.chmod(path, 0444) - change permission of the file / directory in 'path'

Joining LIST of words as paths - using SPLATE '*' operator
>>> ldir
['', 'home1', 'chee', 'dev', 'Kyoto']
>>> os.path.join(*ldir)
'home1/chee/dev/Kyoto'
>>> ldir[1]='/home1'
>>> os.path.join(*ldir)
'/home1/chee/dev/Kyoto'



This will WALK the directory tree:

for (path, dirs, files) in os.walk(path):
    print path
    print dirs
    print files
    print "----"

Here are the results:

c:/python25
['DLLs', 'Doc', 'include', 'Lib', 'libs', 'tcl', 'Tools']
['LICENSE.txt', 'NEWS.txt', 'pylupdate4.exe', 'pyrcc4.exe', 'python.exe', 'pythonw.exe', 'pyuic4.bat', 'README.txt', 'temp.py', 'w9xpopen.exe']
----
c:/python25\DLLs
[]
['bz2.pyd', 'py.ico', 'pyc.ico', 'pyexpat.pyd', 'select.pyd', 'sqlite3.dll', 'tcl84.dll', 'tclpip84.dll', 'tk84.dll', 'unicodedata.pyd', 'winsound.pyd', '_bsddb.pyd', '_ctypes.pyd', '_ctypes_test.pyd', '_elementtree.pyd', '_hashlib.pyd', '_msi.pyd', '_socket.pyd', '_sqlite3.pyd', '_ssl.pyd', '_testcapi.pyd', '_tkinter.pyd']
----
c:/python25\Doc
[]
['Python25.chm']
----
c:/python25\include
[]
['abstract.h', 'asdl.h', 'ast.h', 'bitset.h', 'boolobject.h', 'bufferobject.h', 'cellobject.h', 'ceval.h', 'classobject.h', 'cobject.h', 'code.h', 'codecs.h', 'compile.h', 'complexobject.h', 'cStringIO.h', 'datetime.h', 'descrobject.h', 'dictobject.h', 'enumobject.h', 'errcode.h', 'eval.h', 'fileobject.h', 'floatobject.h', 'frameobject.h', 'funcobject.h', 'genobject.h', 'graminit.h', 'grammar.h', 'import.h', 'intobject.h', 'intrcheck.h', 'iterobject.h', 'listobject.h', 'longintrepr.h', 'longobject.h', 'marshal.h', 'metagrammar.h', 'methodobject.h', 'modsupport.h', 'moduleobject.h', 'node.h', 'object.h', 'objimpl.h', 'opcode.h', 'osdefs.h', 'parsetok.h', 'patchlevel.h', 'pgen.h', 'pgenheaders.h', 'pyarena.h', 'pyconfig.h', 'pydebug.h', 'pyerrors.h', 'pyexpat.h', 'pyfpe.h', 'pygetopt.h', 'pymactoolbox.h', 'pymem.h', 'pyport.h', 'pystate.h', 'pystrtod.h', 'Python-ast.h', 'Python.h', 'pythonrun.h', 'pythread.h', 'py_curses.h', 'rangeobject.h', 'setobject.h', 'sliceobject.h', 'stringobject.h', 'structmember.h', 'structseq.h', 'symtable.h', 'sysmodule.h', 'timefuncs.h', 'token.h', 'traceback.h', 'tupleobject.h', 'ucnhash.h', 'unicodeobject.h', 'weakrefobject.h']
----

To get only files (not directories) on the top level of a directory, say privateFullPath
onlyFiles = [f for f in os.listdir(privateFullPath) if os.path.isfile(os.path.join(privateFullPath, f))]



>subprocess
=============

In order to run a Linux piped command using python, such as ps -A | grep blah,
use the subprocess command two times with subprocess.PIPE to pipe the output to be the input of another

eg
ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout)
ps.wait()

WARNING: arguments above like 'ps' and '-A' MUST NOT have spaces like 'ps ' or ' -A'

Security Issues:
-------------
01 import subprocess
02 import shlex
03  
04  
05 def unsafe(directory):
06     command = "du -sh {}".format(directory)
07     subprocess.call(command, shell=True)
08  
09  
10 def safe(directory):
11     sanitized_directory = shlex.quote(directory)
12     command = "du -sh {}".format(sanitized_directory)
13     subprocess.call(command, shell=True)
14  
15 if __name__ == "__main__":
16     directory = "/dev/null; ls -l /tmp"
17     unsafe(directory)
18     safe(directory)
-------------

in the unsafe() function, the input is not sanitized. The user can put anything.  In this case, the ; ends the du command, then the later command ls -l /tmp, allows people to see contents of /tmp


Another Example - using communicate to capture stdout, stderr
Warning - Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
---
TEST_CMD = './qsub_kyoto.exe'
TEST_ARGS = ' not a-dash -u asdas dfh'
st1 = list()
st1.append(TEST_CMD)
st1.append(TEST_ARGS)
pr1 = subprocess.Popen(st1, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
pout, perr =pr1.communicate()
print pout
print perr
---
The communicate function return a tuple representing stdout, stderr. these can be printed out.
*** Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.
*** Whenever subprocess.PIPE is used for the stdout and stderr, then need to use communicate(). 
Otherwise using wait() will cause deadlocks.
*** Don't use shell=True


Another Case:
I'm surprised you had any success running ls and rmdir without shell=True. ls and rmdir are not programs, they are internal commands within the shell program.
wget, however, is a program. You can use shell=True, but you don't need to. Using shell=False (the default) will save you from having to start a new shell process (which is an expensive operation). When you submit a command with shell=True, you submit the whole command as one string. But when you submit the command with shell=False, you must issue the command as a list of strings, with the command name in the first element of the list, the first argument in the next list element, etc. For example:

wgproc = subprocess.Popen(['wget', '-r', '--tries=10', 'http://fly.srk.fer.hr/', '-o', 'log'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
(standardout, junk) = wgproc.communicate()



Example - using stdout to capture output to a file:
ofile = open("/tmp/out.txt", "wb")
change=subprocess.Popen(["echo","0"], stdout=ofile)
ofile.close() 

--------------

If you need to execute a command with a timeout, use the timeout argument:
try:
    out_bytes = subprocess.check_output(['cmd','arg1','arg2'], timeout=5)
except subprocess.TimeoutExpired as e:
    ...
Normally, commands are executed without the assistance of an underlying shell (e.g., sh, bash, etc.). Instead, the list of strings supplied are given to a low-level system command, such as os.execve(). If you want the command to be interpreted by a shell, supply it using a simple string and give the shell=True argument. This is sometimes useful if you’re trying to get Python to execute a complicated shell command involving pipes, I/O redirection, and other features. 

For example:
out_bytes = subprocess.check_output('grep python | wc > out', shell=True)

Be aware that executing commands under the shell is a potential security risk if arguments are derived from user input. The shlex.quote() function can be used to properly quote arguments for inclusion in shell commands in this case.





Commands
===========
exec(open('<python file>').read())   - executes a python script file *.py
glob.glob(path/*.txt)                - grabs a list of filenames
interpreter = re.match('^#!(.*)', line).group(1)
        Matches the regular expression in the "line" string, then chooses the first group.
m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
m.group(0)       # The entire match
>>>'Isaac Newton'
m.group(1)       # The first parenthesized subgroup.
>>>'Isaac'
m.group(2)       # The second parenthesized subgroup.
>>>'Newton'
m.group(1, 2)    # Multiple arguments give us a tuple.
>>>('Isaac', 'Newton')

import sys; print sys.version      # version of python


"Be careful with exec and eval in Python"
http://lucumr.pocoo.org/2011/2/1/exec-in-python/


Command line Options
=======================
# Get the total number of args passed to the demo.py
total = len(sys.argv)
 
# Get the arguments list 
cmdargs = str(sys.argv)

# sys.argv values CAN BE CHANGED
print sys.argv[1]
sys.argv[1] = 'blah'
print sys.argv[1]


1. Using getopt
>>> s = '--condition=foo --testing --output-file abc.def -x a1 a2'
>>> args = s.split()
>>> args
['--condition=foo', '--testing', '--output-file', 'abc.def', '-x', 'a1', 'a2']
>>> optlist, args = getopt.getopt(args, 'x', [
...     'condition=', 'output-file=', 'testing'])
>>> optlist
[('--condition', 'foo'), ('--testing', ''), ('--output-file', 'abc.def'), ('-x', '')]
>>> args
['a1', 'a2']
----
 try:
      opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])
   except getopt.GetoptError:
      print 'test.py -i <inputfile> -o <outputfile>'
      sys.exit(2)
Note: The getopt assumes options are either single '-x' or long names '--xxxxxxx'
The notation for single is ":" and for long names is "="


2. Using optparse
Example:
----------------
import optparse
if __name__ == '__main__':
    entry = optparse.OptionParser(description="KyotoLSF to PBSPro Translator",
                                 prog="Converter", version="1.0",
    usage="python %prog.py -ug <group> -W <walltime>  -A <node-core specs> ")
#    entry.add_option("--source_files_path", "-s", default="")
    entry.add_option("-u", "--user_group",  type="string", action="store", dest="OPT_UG")
    options, arguments = entry.parse_args()
----------------



Octo.py
========
Ref: http://code.google.com/p/octopy/

MapReduce implementation in Python.

Put octo.py on all of the machines you want to use. 
On the machine you will use as a server (with ip address <ip>), also install triangular.py, and then execute:
     python octo.py server triangular.py &

On each of your clients, run
     python octo.py client <ip> &

You can try this out using the same machine to run the server process and one or more client processes, of course.


Unix Script
=============
Example Script:

<code>
#!/usr/bin/env python
import sys
import subprocess

sUsage = 'Usage: sudo <command> username'

print "**** users_du script"
print "**** prints the usages for each users"
print sUsage

# Check number of arguments
if len(sys.argv) == 1:                   # sys.argv[0] is the actual command
 print "Disk usage for all users"
 proc_ls = subprocess.Popen(['ls', '/home'], stdout=subprocess.PIPE)

 for line in proc_ls.stdout.readlines():
  sdir = '/home/' + line.rstrip()
  proc_du = subprocess.Popen(['sudo', 'du', '-s', sdir], stdout=subprocess.PIPE)
  for line_du in proc_du.stdout.readlines():
   print line_du
   stop

elif len(sys.argv) == 2:
 print "Disk usage for user ", sys.argv[1]
 sdir = '/home/' + sys.argv[1]
 proc_du = subprocess.Popen(['sudo', 'du', '-s', sdir], stdout=subprocess.PIPE)
 for line_du in proc_du.stdout.readlines():
  print line_du

else:
 sys.exit(0)


print "End of Script"
</code>


1. use header
#!/usr/bin/env python

2. Useful packages
import sys   # system commands            
import optparse     # allow script to have various options
import subprocess   # shell scripting features


print
======
Combininb strings and numbers
Eg.
 xx=3
 print ("hello" + str(xx))


Format specifications:
's'  String format. This is the default type for strings and may be omitted.
'b'  Binary format. Outputs the number in base 2.
'c'  Character. Converts the integer to the corresponding unicode character before printing.
'd'  Decimal Integer. Outputs the number in base 10.
'o'  Octal format. Outputs the number in base 8.
'x'  Hex format. Outputs the number in base 16, using lower- case letters for the digits above 9.
'X'  Hex format. Outputs the number in base 16, using upper- case letters for the digits above 9.
'n'  Number. This is the same as 'd', except that it uses the current locale setting to insert the appropriate number separator characters.
None  The same as 'd'.

'e'  Exponent notation. Prints the number in scientific notation using the letter ‘e’ to indicate the exponent. The default precision is 6.
'E'  Exponent notation. Same as 'e' except it uses an upper case ‘E’ as the separator character.
'f'  Fixed point. Displays the number as a fixed-point number. The default precision is 6.
'F'  Fixed point. Same as 'f'.
'g'  

General format. For a given precision p >= 1, this rounds the number to p significant digits and then formats the result in either fixed-point format or in scientific notation, depending on its magnitude.

The precise rules are as follows: suppose that the result formatted with presentation type 'e' and precision p-1 would have exponent exp. Then if -4 <= exp < p, the number is formatted with presentation type 'f' and precision p-1-exp. Otherwise, the number is formatted with presentation type 'e' and precision p-1. In both cases insignificant trailing zeros are removed from the significand, and the decimal point is also removed if there are no remaining digits following it.

Positive and negative infinity, positive and negative zero, and nans, are formatted as inf, -inf, 0, -0 and nan respectively, regardless of the precision.

A precision of 0 is treated as equivalent to a precision of 1. The default precision is 6.
'G'  General format. Same as 'g' except switches to 'E' if the number gets too large. The representations of infinity and NaN are uppercased, too.
'n'  Number. This is the same as 'g', except that it uses the current locale setting to insert the appropriate number separator characters.
'%'  Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign.
None  The same as 'g'.



Check Text Format Newline CLRF
================================
To check whether a text file has Windows /r/n or Unix /n or Mac /r line endings, do this:

f = open('myfile.txt', 'U')
f.readline()  # Reads a line
# The following now contains the newline ending of the first line:
# It can be "\r\n" (Windows), "\n" (Unix), "\r" (old Mac OS).
# If no newline is found, it contains None.
print repr(f.newlines)


WARNINGS
=========
A few things that Python are different to others and are counter-intuitive.
 
1. Array slicing, a[2:5] means a[2], a[3], a[4] but NOT a[5]. Better to think that indices are actually on the edge of number line, and contents in the middle.

2. Python "=" are NOT copy operators. For example
  a=7
  b=a
  b=3
  print a   .... gives 3, not 7

  To copy variable, use the copy method, eg
  b = a.copy()







Sourcing Script within a python file
======================================
I have a common script that needs to be called from every app definition’s start.py. What is the simple way of doing this. There are no functions inside this script, it is a plain script that reads some environment variables and outputs some information to the job directory before the job is completed.
Did a google search for this and most point to using os.system call to start a new python execution and run this script. Or define a function, use import statement to call the script and then call the function.
Is there a way to just source the script in the same python environment as start.py
I am trying to avoid duplicate code being used in multiple app definitions.


os.system is fine in simple cases. In more advanced cases (where you want more control programmatically) use subprocess.
<code>
    if os.path.exists(“/path/to/script"):
        command = “/path/to/script"
        script = subprocess.Popen(command, env=os.environ, shell=True, stdout=subprocess.PIPE)
        for line in script.stdout.readlines():
              print line
        script.wait()
</code>

In case you really want to “source” a script you can do things like
<code>
mpivars_script = os.environ['mpipath'] + '/bin64/mpivars.sh'
p = subprocess.Popen(". %s; env" % mpivars_script, stdout=subprocess.PIPE, shell=True)
stdout = p.communicate()[0]
env = dict((line.split("=", 1) for line in stdout.splitlines()))
os.environ.update(env)
</code>
But if you want to re-use your Python code the best is to write modules an import them with import.

 

Python openedx
=================
https://github.com/numerical-mooc/numerical-mooc


iPython /Jupyter
=================
to start:   ipython notebook

in markdown cell, to add a image to the notebook, need the "files/" to prepend to relative path. Eg.
![Image](files/figures/glider_forces.png)

 

 
Symbolic Math Python
=====================

import sympy        
from sympy.utilities.lambdify import lambdify

Declare a variable to be symbolic"
   x, nu = sympy.symbols('x nu')
WARNING - these symbols should not be variables that are intended to have a value assigned. Eg if nu=0.3 is defined, then it is a constant, NOT a symbolic variable.

Declare a maths expression or formula
   y = sympy(x)**3

Differentiate symbolically
   dydx = y.diff(x)

Lambdify - makes the symbolic expression into a function
- creating the lambda function
     yfunc = lambdify((x, ..), y)
- using the function
     yfunc(2.3)     --- evaluating y(x=2.3)



Printing Math Formula
=======================
from sympy import *
... has functions like: symbols, latex

ref: http://oneau.wordpress.com/2011/12/25/latex-sympy/
>>> x = Symbol("x")
>>> a =  1/( (x+2)*(x+1) )
>>> latex(a)
\frac{1}{\left(x + 1\right) \left(x + 2\right)}

PrettyPrint - to print latex equations onto png or dvi or terminal


from sympy.printing import printing
preview(str, outputTexFile='sample.tex')


   str = "line 1 \\\\" +" line 2\\\\ " + " line 3\\\\ "
 x,y=symbols('x,y')
   str = str + '$'+latex(x**2 + y)+'$'
   preview(str, outputTexFile='sample.tex')
This will produce the lines
    line 1
 line 2
 line 3
 x^2 + y 

preview(str, output='pdf', viewer='notepad')


Installing Packages
=====================
Instruction for windows
1. Download python packages such as numpy, etc
2. Unpack in any directory
3. Open up Command Prompt
4. Set the python path, such as: (may be different for your installation)
      set path=%path%;C:\python27
5. Change directory to where the packages like numpy, were unpacked.
6. To install the packages, typically (but may not be for all cases):
       python setup.py install
      

 
Linux package install

Numpy - 
Scipy - http://www.scipy.org/install.html
MPI4py - http://www.mpi4py.scipy.org/docs/usrman/tutorial.html# 


>Packages for Machine Learning
================================

scipy - http://www.scipy.org/install.html

numpy - mainly useful for its N-dimensional array objects

matplotlib - 2D plotting library producing publication quality figures

pandas - Python data analysis library, including structures such as dataframes
http://pandas.pydata.org/pandas-docs/stable/10min.html

sklearn - the machine learning algorithms used for data analysis and data mining tasks
http://scikit-learn.org/stable/install.html 
http://scikit-learn.org/stable/tutorial/basic/tutorial.html


Canopy - able to use Package Manager




Zipfile and Datetime
=====================
import os
import zipfile
import datetime

# Getting date and appending .zip
today = datetime.date.today()
zipfilename = str(today) + '.zip'   # str(today) format as "2015-02-16"
print zipfilename

# get files in directory
files = os.listdir('.')
#print files

# ZipFile is a context manager and can exit safely
with zipfile.ZipFile(zipfilename, 'w') as myzip:
 for ifile in files:           # looping through each file in the directory
  if ifile[-4:] == '.pdf':   # consider only *.pdf files
   print ifile
   myzip.write(ifile)      # add each *.pdf file into zip file.


A very good list of examples are in:
(http://www.pythonforbeginners.com/basics/python-datetime-time-examples/)
import time
import datetime

print "Time in seconds since the epoch: %s" %time.time()
print "Current date and time: " , datetime.datetime.now()
print "Or like this: " ,datetime.datetime.now().strftime("%y-%m-%d-%H-%M")

Formats fmt in datetime.date.today().strftime(fmt)
%Y Current year
%B Month of year: 
%W Week number of the year: "
%w Weekday of the week: 
%j Day of year: 
%d Day of the month : 
%A Day of week: 

for a time of 16:13:37
str(datetime.datetime.now()).split(" ")[1].split(".")[0].replace(":", "")
=> 161337

PyCharm Editor
=====================

How to disable auto-save:

    Go to File > Settings (Ctrl+Alt+S).
    Go to Appearance & Behavior > System Settings.
    Make sure the two are unchecked:
        Save files on frame deactivation
        Save files automatically if application is idle for x sec.
    Go to Editor > General > Editor Tabs
    Put a checkmark on "Mark modified files with asterisk"
    (Optional but recommended) Under "Tab Closing Policy", select "Close non-modified files first". You may also want to increase the number of allowed tabs.
    Click Apply > OK.

How to Change versions of Python Interpreter
- Click on the desired project on the left nav area
- Click File -> Settings. 
- In the Settings dialog, click: Project -> Project Interpreter
- On the main page of the dialog, click on the correct project to change
- In the Project Interpreter: select or add the correct Python version.


Error: Missing module sys - "import sys" module not found
Solution: File -> "Invalidate Caches / Restart"

To Show line numbers, go to:
File - Settings -Editor - General - Appearance


Python Logging module
==========================
Best shown with example below (from http://www.onlamp.com/pub/a/python/2005/06/02/logging.html)

-----------------------
#!/usr/bin/env python

import logging
import auxiliary_module

#create logger
logger = logging.getLogger("simple_example")
logger.setLevel(logging.DEBUG)
#create console handler and set level to error
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
#create file handler and set level to debug
fh = logging.FileHandler("spam.log")
fh.setLevel(logging.DEBUG)
#create formatter
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - 
   %(message)s")
#add formatter to ch and fh
ch.setFormatter(formatter)
fh.setFormatter(formatter)
#add ch and fh to logger
logger.addHandler(ch)
logger.addHandler(fh)

#"application" code
logger.debug("debug message")
logger.info("info message")
logger.warn("warn message")
logger.error("error message")
logger.critical("critical message")

logger.info("creating an instance of auxiliary_module.Auxiliary")
a = auxiliary_module.Auxiliary()
logger.info("created an instance of auxiliary_module.Auxiliary")
logger.info("calling auxiliary_module.Auxiliary.do_something")
a.do_something()
-----------------------
#!/usr/bin/env python

import logging

#create logger
module_logger = logging.getLogger("spam_application.auxiliary")


class Auxiliary:
    def __init__(self):
        self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary")
        self.logger.info("creating an instance of Auxiliary")
    def do_something(self):
        self.logger.info("doing something")
        a = 1 + 1
        self.logger.info("done doing something")

def some_function():
    module_logger.info("received a call to \"some_function\"")
------------------------

There two python code above shows two different things together
1) Multiple handlers for different error levels 
2) Multiple modules logging

The second code is a module (Auxilliary) called by the first code.

Naming the 3 loggers:
logger = logging.getLogger("simple_example")   # for main script
module_logger = logging.getLogger("spam_application.auxiliary")    # for module
self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary") # for class

Two handlers
ch = logging.StreamHandler()
fh = logging.FileHandler("spam.log")

... which handle to log levels
ch.setLevel(logging.ERROR)
fh.setLevel(logging.DEBUG)


Installing a Second Python
============================
Ref: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

Install the following:
yum groupinstall "Development tools"
yum install zlib-devel 
yum install bzip2-devel 
yum install openssl-devel 
yum install ncurses-devel 
yum install sqlite-devel 
yum install readline-devel 
yum install tk-devel 
yum install gdbm-devel 
yum install db4-devel 
yum install libpcap-devel 
yum install xz-devel

Configure:
Use the following configure switches

Use unicode for extra compatibility.
2.7> --enable-unicode=ucs4 
3.2>--with-wide-unicode

Compile as shared library, as most python dist, otherwise not compatible with some code.
Add as configure flag:
LDFLAGS="-Wl,-rpath /usr/local/lib"

Alternatively edit /etc/ld.so.conf and add line: 
 /usr/local/lib.
Then make the dynamic linker aware:
 /sbin/ldconfig

Installing:
 make altinstall

Summary:
# Python 2.7.6:
wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
tar xf Python-2.7.6.tar.xz
cd Python-2.7.6
./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
 
# Python 3.3.5:
wget http://python.org/ftp/python/3.3.5/Python-3.3.5.tar.xz
tar xf Python-3.3.5.tar.xz
cd Python-3.3.5
./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall


Python Package Manager - Setuptools and pip
==============================================
pip is used to manage and install python packages for your installation.
Setuptools need to be installed first before using it to install pip.


# First get the setup script for Setuptools:
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
 
# Then install it for Python 2.7 and/or Python 3.3:
python2.7 ez_setup.py
python3.3 ez_setup.py
 
# Now install pip using the newly installed setuptools:
easy_install-2.7 pip
easy_install-3.3 pip
 
# With pip installed you can now do things like this:
pip2.7 install [packagename]
pip2.7 install --upgrade [packagename]
pip2.7 uninstall [packagename]




Isolate Python Environment using virtualenv
===============================================
Ref: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

# Install virtualenv for Python 2.7 
pip2.7 install virtualenv

# Python 3.3 already has pyvenv equivalent

#create a sandbox called my27project:
virtualenv-2.7 my27project
 
# create a sandbox called my33project:
pyvenv-3.3 my33project
 
# the system Python interpreter should still be default:
python --version
# Should show Python 2.6.6 for Centos 6
 
# to use my27project sandbox:
source my27project/bin/activate
python --version
# Should show Python 2.7.6
deactivate
 
# to use my33project sandbox :
source my33project/bin/activate
python --version
# Should show Python 3.3.5
deactivate


@ Declarators
===============
One out of many example showing the use of declarators:
----------------------
class myDecorator(object):

    def __init__(self, f):
        print "inside myDecorator.__init__()"
        f() # Prove that function definition has completed

    def __call__(self):
        print "inside myDecorator.__call__()"

@myDecorator
def aFunction():
    print "inside aFunction()"

print "Finished decorating aFunction()"

aFunction()
When you run this code, you see:

inside myDecorator.__init__()
inside aFunction()
Finished decorating aFunction()
inside myDecorator.__call__()
----------------------





Python uploading to Gdrive
=============================
See NotesPythonGoogleAPI


Python Writing Blog 
=====================
See NotesPythonGoogleAPI



Troubleshooting
=================

Message:
ImportError: No module named
Occurance:
Python programs using modules/packages, typically when programs have import statements.
Cause:
The module's package may have been installed but the path was not found.
If the package was not installed, then install using pip or some other tools.
Solution:
On linux, set the environment variable like below, in Bash:
export PYTHONPATH="/usr/local/lib/python2.7/site-packages"


Miscellaneous
=================
os.path.expanduser("~") - in Windows8, returns C:\Users\<username>
[''] and array of empty strings is counted as len 1. Mistake to assume this object has ZERO length
for ii in range (1,N) - this loops up to N-1 only.

To use New Python 2.6 features on Python 2.4:
from __future__ import with_statement


UnitTest
==========
http://pythontesting.net/start-here/
https://docs.python.org/2/library/unittest.html
http://docs.python-guide.org/en/latest/writing/tests/
http://pyunit.sourceforge.net/pyunit.html


Example:
<code>
import unittest

class TestAoption(unittest.TestCase):
    """ TestAoption

    """
    def test_dummy(self):
        """Dummytest

        :return:
        """
        self.assertEqual("hello", "hello")

if __name__ == '__main__':
    unittest.main()
</code>

To run:
1.  python test_Aoption_unittest.py
2. 


umask in python
==================
import os
import stat

# Define file params
fname = '/tmp/myfile'
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL  # Refer to "man 2 open".
mode = stat.S_IRUSR | stat.S_IWUSR  # This is 0o600 in octal and 384 in decimal.

# For security, remove file with potentially elevated mode
try:
    os.remove(fname)
except OSError:
    pass

# Open file descriptor
umask_original = os.umask(0)
try:
    fdesc = os.open(fname, flags, mode)
finally:
    os.umask(umask_original)

# Open file handle and write to file
with os.fdopen(fdesc, 'w') as fout:
    fout.write('something\n')

If the desired mode is 0600, it can more clearly be specified as the octal number 0o600. Even better, just use the stat module.

Even though the old file is first deleted, a race condition is still possible. Including os.O_EXCL with os.O_CREAT in the flags will prevent the file from being created if it exists due to a race condition. This is a necessary secondary security measure to prevent opening a file that may already exist with a potentially elevated mode. In Python 3, FileExistsError with [Errno 17] is raised if the file exists.

Failing to first set the umask to 0 can lead to an incorrect mode (permission) being set by os.open. This is because the default umask is usually not 0, and it may be applied to the specified mode. For example, if my original umask is 2, and my specified mode is 0o222, if I fail to first set the umask to 0, the resulting file can instead have a mode of 0o220, which is not what I wanted. Per man 2 open, the mode of the created file is mode & ~umask.

The umask is restored to its original value as soon as possible. This getting and setting is not thread safe, and a threading.Lock must be used if necessary.

Coverage
===========
coverage.py
https://coverage.readthedocs.io/en/coverage-4.4.1/

- to install: pip install coverage
- create a program such as a.py that will take arguments, eg:
    python a.py hello
- run coverage: 
    coverage run a.py hello
- make report: (reports should appear in same directory)
    coverage report -m
    coverage html       (in ./htmlcov)
    coverage xml


Codacy
==========

- to install in env with python already: pip install codacy-coverage
- Go to Codacy website about setup test coverage, and get token.
- Set token on Windows, on the same CMD terminal as before,
    set CODACY_PROJECT_TOKEN=blah
- Perform coverage using codacy.
    python-codacy-coverage -r coverage.xml

Message:
(C:\Users\chee\Anaconda2) D:\My Box Files\workspace\gittest>set CODACY_PROJECT_TOKEN=blahhhhhhh

(C:\Users\chee\Anaconda2) D:\My Box Files\workspace\gittest>python-codacy-coverage -r coverage.xml
Traceback (most recent call last):
  File "c:\users\chee\anaconda2\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\users\chee\anaconda2\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\chee\Anaconda2\Scripts\python-codacy-coverage.exe\__main__.py", line 9, in <module>
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\__init__.py", line 6, in main
    return reporter.run()
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\reporter.py", line 198, in run
    args.commit = get_git_revision_hash()
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\reporter.py", line 41, in get_git_revision_hash
    return subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode("utf-8").strip()
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 212, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 640, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified


Solution:
The error is because Codacy at this point does not run on Windows.
Tested Successful on Linux Centos6.4


Security - Bandit
===================
Bandit package
https://pypi.python.org/pypi/bandit/1.0.1


- install: pip install bandit
- to run:   bandit -r /path/to/code

xTechNotes - Technical Computing Programming Notes

Tuesday, October 30, 2007

Notes CSS

Wednesday, October 24, 2007

Notes Windows Power Shell

Thursday, August 02, 2007

Notes Fortran

Thursday, July 26, 2007

Notes Video Capture

NotesDVD

Saturday, July 21, 2007

Building own PC

Thursday, June 14, 2007

NotesWeb

Wednesday, May 16, 2007

NotesAudio

Saturday, July 22, 2006

Notes Linux

Tuesday, January 31, 2006

NotesHTMLtags

NotesPython

Translate

Followers

Search This Blog

Blog Archive

Support xTechNotes Blog

Keywords

Related Books

Tech Blog Links

Other Links

Total Pageviews

Tuesday, October 30, 2007

Wednesday, October 24, 2007

Thursday, August 02, 2007

Thursday, July 26, 2007

Saturday, July 21, 2007

Thursday, June 14, 2007

Wednesday, May 16, 2007

Saturday, July 22, 2006

Tuesday, January 31, 2006

Subscribe To

Translate

Followers

Search This Blog

Blog Archive

Support xTechNotes Blog

Keywords

Related Books

Tech Blog Links

Other Links

Total Pageviews