NotesPython

Contents
=========
References
Style Guide
>Doxygen
Introduction
Compile
Tkinter - Python's GUI
Syntax
Data Types
Umask and file creation
Operators
Strings 
Lists
Collections - defaultdict
Math, Array, builtin functions
Flow Control
Functions, Modules, Append path
>Arguments
Global vars
Classes 
>Inheritance
>classmethod, staticmethod
Exceptions
>Passing Exceptions to outer functions
>Exception v2 vs v3
>Signals
Differences between Python 2.x vs Pyhon 3.x
>Exceptions
Importing 
File I/O
Writing CSV file
OS FileSystem and shell commands
>subprocess
Commands
Command line Options
Octo.py
Unix Script
Print
Check Text Format Newline CLRF
WARNINGS
<more complex things>
Sourcing Script within a python file
Python openedx
iPython /Jupyter
Symbolic Math Python
Printing Math Formula
Installing Packages
>Packages for Machine Learning
Zipfile and Datetime
PyCharm Editor
Python Logging module
Installing a Second Python
Python Package Manager - Setuptools and pip
Isolate Python Environment using virtualenv
@ Declarators
Python uploading to Gdrive
Python Writing Blog 
Troubleshooting
Miscellaneous
UnitTest
umask in python
Coverage
Codacy
Security - Bandit




References
===========
http://www.poromenos.org/tutorials/python
http://docs.python.org/tutorial/index.html
http://docs.python.org/py3k/tutorial/index.html
http://infohost.nmt.edu/tcc/help/pubs/tkinter/minimal-app.html - Tkinkter reference
http://www.tutorialspoint.com/python/python_gui_programming.htm
http://www.dreamsyssoft.com/python-scripting-tutorial/shell-tutorial.php

Library Ref: http://docs.python.org/2.7/library/
Language Ref: http://docs.python.org/2.7/reference/index.html

Style Guide
=============
https://www.python.org/dev/peps/pep-0008/



Example of docstring - using Triple quotes 

"""Example Google style docstrings.

This module demonstrates documentation as specified by the `Google Python 
Style Guide`_. Docstrings may extend over multiple lines. Sections are created
with a section header and a colon followed by a block of indented text.

Example:
    Examples can be given using either the ``Example`` or ``Examples``
    sections. Sections support any reStructuredText formatting, including
    literal blocks::

        $ python example_google.py

Section breaks are created by resuming unindented text. Section breaks
are also implicitly created anytime a new section starts.

Attributes:
    module_level_variable1 (int): Module level variables may be documented in
        either the ``Attributes`` section of the module docstring, or in an
        inline docstring immediately following the variable.

        Either form is acceptable, but the two should not be mixed. Choose
        one convention to document module level variables and be consistent
        with it.

.. _Google Python Style Guide:
   http://google.github.io/styleguide/pyguide.html
"""


>Doxygen
============
To use Doxygen, the python code must be like the following:
<code>
1 """@package docstring
2 Documentation for this module.
3 
4 More details.
5 """
6 
7 def func():
8  """Documentation for a function.
9 
10  More details.
11  """
12  pass
13 
14 class PyClass:
15  """Documentation for a class.
16 
17  More details.
18  """
19 
20  def __init__(self):
21  """The constructor."""
22  self._memVar = 0;
23 
24  def PyMethod(self):
25  """Documentation for a method."""
26  pass
</code>









Introduction
=============
Phython - What is it?
 - interpreted (no compilation)
 - interactive
 - object-oriented
 - compared with Tcl, Perl, Scheme, Java

Python - attributes
 - power and clear syntax
 - has modules, classes, exceptions, dynamic data types
 - interfaces to system calls, libraries,
 - interfaces to windowing systems X11, Motif, Tk, Mac, MFC

 - built-in modules can be written in C/C++
 - useable as an extension language for applications that need a programming interface.

Running Python
- run python script file *.py by:
   python.exe <filename>
... any file I/O assumes files are in the same directory.
- run python interactively using the GUI or Text console.
- run Python with Enthought Canopy editor
... any file I/O assumes files are in C:\users
... use os.getcwd() to find out current working dir
... use os.chdir(r'...') to change working directory
- run python without saying python:
eg file is pyVersion.py
--------------------
#/opt/pbs/default/python/bin/python
import sys
print (sys.version)
-------------------
To run this, just type:  ./pyVersion.py


Modules
- module file has python definitions and statements and called <modfile>.py
- to include this module to use in script or interactively:
    import <modfile>
- to use any functions, say foo, in <modfile>.py,
    <modfile>.foo
- to import specific functions only
    from <modfile> import foo
- to import everything
    from <modfile> import *

Run module file
    python <modfile>.py <arguments>

the module code has to include the following at the end 
----
if __name__ == "__main__":
    import sys
    fib(int(sys.argv[1]))
----
you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is executed as the “main” file:
So the example above will run the fib() function in <modfile>.py with arguments sys.argv[1] using the <arguments>


The Python interpreter reads a source file, and executes all of the code found in it. It will define a few special variables, eg __name__. If the python interpreter is running that module *.py as the main program, it sets __name__ to be "__main__". If this file is being imported from another module, __name__ will be set to the module's name.

The reasons for doing this is that sometimes you write a module (a .py file) where it can be executed directly. Additionally, it can also be imported and used in another module. By checking <if __name__ == "__main__":>, that code only execute when the module is run as a program and not run when the module is being imported into another code.


Run python script file (this is like module since it can contain function defs, but also have python executable statements)
 %run <filename>.py

B=A .....B is not a copy. It is a reference.


fileinput package:
----------
import fileinput
for line in file.input():
    print line

This will print out the lines of the files (eg data.txt) given to this python script. It can be run in TWO ways:
i) ./myprogram.py  data.txt
ii) ./myprogram.py < data.txt      (file indirection)




Compile
===========
$ python
>>> import py_compile
>>> py_compile.compile('server.py')
>>> ^D
$ python ./server.pyc




Tkinter - Python's GUI
======================

1. To use Tkinter, need
       from Tkinter import *

2. Set-up Root Window: 
 Tk()                           ### create Top-level window
 Tk.mainloop                    ### create Top-level window()

example:
  root = Tk()           ### create Top-level window
  myapp = MyApp(root)
  root.mainloop()  ### starts the EVENTLOOP to  listen to events
  root.destroy()        ### closes and exits this window called root


3. Containers:
 Frame(parent)                   ### starts a frame belonging to Parent
 Frame.pack()                    ### display the frame , geometry manager
 Frame.grid()                    ### geometry manager
 Frame.place()                   ### geometry manager



example:
   set up containers using:
       container1 = Frame(parent)

   ... where parent is the widget container1 belongs to.

   displaying containers, use:
       container1.pack()

   other geometry managers besides PACK, are GRID and PLACE


4. Buttons:
 Button(parent, attr=value,)        ### starts buttons and sets attributes
 Button.pack()                      ### displays the Button
 Button.configure(attr=value, ...)  ### sets some attributes
 Button["attr"]="value"             ### sets some attributes

   where attr = { text, background }


example:
   set up buttons:
       button1 = Button(parentContainer)

   display buttons:
       button1.pack()


5. Event Handling - i) Event Binding  ii) Command Binding
   Syntax:
 widget.bind( self, event, event_type_name, event_handler_name )

   event_type_name = { "<Button-1>", ....
 
   

Syntax
=========
Comments - # sign
Indentation - to specify blocks
Indentation - expected when statement end in colon ":" 
Line Continuation -  \



Data Types
===========
List, tuples, dictionaries, sets (sets library)
First (N items) array item index is 0, or -N
Last (N items) array item index is N-1 or -1

 List - 1D arrays, can also have List of Lists
Eg. list[:] - indexing single element
lstSample = [1, ["another", "list"], ("a", "tuple")]
lstList = ["List item 1", 2, 3.14]
lstList[0] = "List item 1 again"
lstList[-1] = 3.14
lstList[-2] =2
lstList[-3] = "List item 1 again"

Eg. list[i:j] - indexing / SLICING a sequence
think of the following sequence, where index are at the edge on in the middle.
0             1     2            3    -> Counter
  List Item 1    2    3.14000000      -> element sequence

>>> print lstList[:]          -> everything
['List item 1', 2, 3.1400000000000001]
>>> print lstList[0:2]        -> from right of index 0 to before index 2
['List item 1', 2]
>>> print lstList[-3:-1]      -> from right of index -3 to before index -1
['List item 1', 2]
>>> print lstList[1:]         -> from right of index 1 to the end
[2, 3.14]

Eg. list[i;j;k] - sequence with steps
>>> "hello"[::-1]             -> the whole sequence but with backward (-1) step
"olleh"


Eg. List Manipulation
lst1 = [1, 2, 3]
lst2 = [3, 4, 5]
>>> print [x * y for x in lst1 for y in lst2]
[3, 4, 5, 6, 8, 10, 9, 12, 15]
>>> print [x for x in lst1 if 4 > x > 1]
[2, 3]


 Tuples - immutable 1D array
Eg
tplTuple = (1, 2, 3)


 Dictionaries - hash tables or associative arrays
dicDictionary = {"Key 1": "Value 1", 2: 3, "pi": 3.14}
dicDictionary["pi"] = 3.15

Adding to dictionary:
default_data['item3'] = 3
default_data.update({'item4': 4, 'item5': 5})

Deleting item from dictionary
del dicDictionary["pi"]      # this will delete the "pi" key-value pair

dictionary can be intersected with list - see Lists section


Oct vs Int:
Python 2.6+
>>> oct(9)
'011'
>>> int(0o010)
8
>>> int(0o011)
9

Python 2.5-
>>> oct(9)
'011'
>>> int('010',8)
8
>>> int('011',8)
9


Umask and file creation
=============================
https://stackoverflow.com/questions/10291131/how-to-use-os-umask-in-python

os.stat
https://stackoverflow.com/questions/5337070/how-can-i-get-a-files-permission-mask
http://effbot.org/librarybook/stat.htm
 
https://stackoverflow.com/questions/5624359/write-file-with-specific-permissions-in-python

>>> import os
>>> os.getcwd()
'/home/chee/dev/Kyoto/tmp'
# Using default umask
>>> ff=open('pumask.txt','w')  
>>> ff.close()
>>> umask0002=os.umask(0066)
>>> print umask0002
2
# Using the 066 umask
>>> ff=open('pmask0066.txt','w')
>>> ff.close()
>>> umask0066=os.umask(umask0002)
>>> print umask0066
54
# using the reverted umask
>>> ff=open('pmask0002a.txt','w')
>>> ff.close()

-rw-rw-r--  1 chee chee       0 Dec 22 01:08 pumask.txt
-rw-------  1 chee chee       0 Dec 22 01:37 pmask0066.txt
-rw-rw-r--  1 chee chee       0 Dec 22 01:49 pmask0002a.txt


Or using os.open
------

import os
fd = os.open('/path/to/file', os.O_WRONLY, 0o600)
myFileObject = os.fdopen(fd)
myFileObject.write(...)
myFileObject.close()


Operators
==========
=  assignment
+=  add numbers or strings
-=  subtract
==  equality test
1 < a < 3 checks that a is both less than 3 and more than 1
del  delete|remove|clear|nullify variables, array elements or arrays


conditional operators:
==
!=
<>
>
<
>=
<=

Logical operators:
and
or
not = negate operator


 Variable Name assignment
Eg. 
Listt =[1,2]
fnVariable = len         # len is length function
print fnVariable(Listt)  # prints length of Listt



Strings 
=========
"blah" or 'blah' or "this 'is' blah" or 'this "is" blah'
''' this is 
multi-line string'''
"""this is 
multi-line string"""

unicode -> u"This is unicode string"


Find / Replace in Strings:
sout = "blah blah job_ID"
ii = sout.find("job_ID", beg=0, end=len(string))
-> returns index position of text found, or -1 if text not found.

sout.replace("$JOB_ID", jid_num)
-> replaces any "$JOB_ID" with string in jid_num


Value in Strings
eg. print "Name: %s\nNumber: %s\nString: %s" % (class.name, 3, 3 * "-")
Name: Poromenos
Number: 3
String: ---

Parse string:
In [8]: lhs, rhs = "2.7.0_bf4fda703454".split("_", 1)
In [9]: lhs
Out[9]: '2.7.0'
In [10]: rhs
Out[10]: 'bf4fda703454'


>>> "2.7.0_bf4fda703454".partition('_')
('2.7.0', '_', 'bf4fda703454')

>>> "shazam".partition("_")
('shazam', '', '')


"This is an example of \n\
   end of line string. But last one is \
   like that"

Substrings
   wordd = 'Hello World"
   wordd[0:2]        - 'Hel'
   wordd[:2]         - first two characters
   wordd[2:]         - from the first two to the last character.
   '<'+wordd*5+'>'   - concatenate with +, also repeat 5 times with *5.

   word = 'HelpA'
   word[-1]    'A'     - from the right end
   word[-2]    'p'
   word[-2:]   'pA'
   word[:-2]  'Hel'

Literal strings:   r'hello\world'   - treats the slash as a slash character, not something else.


Arguments in string:
sentence = "Hello {0}"
print sentence.format("Ted")    -> "Hello Ted"
print "Hello {0}".format("Ted")     -> "Hello Ted"
people="Ben"
place="Barn"
sentence = "I went .. {0} with {1}".format(people. placce)

String functions:
sentence.capitalize()
sentence.lower()
sentence.upper()
sentence.swapcase()
sentence.title()
sentence.count()
sentence.endswith()
sentence.isalnum()
sentence.isalpha()
sentence.isdigit()
sentence.islower()
sentence.isupper()
sentence.isspace()
sentence.lstrip()
sentence.rstrip()
sentence.strip()
sentence.replace()
sentence.split()
sentence.join()




Lists
======
a = ['spam', 'eggs', 100, 1234]
Indices start from 0, like strings
list.append()        - adds element to the end of list
list.pop()           - remove last element from list
list.extend()           - add last element to list

To have first in - first out, need deque:
    from collections import deque
the following is how to use the deque class.
myQ = deque([1,2,3])
myQ.append()         - adds element to the end of list
myQ.popleft()        - removes first element from the list.

mylist[-1]           - gives the last item on the list
mylist.index("blah") - returns index
"z" in my_list       - returns T/F to check if element is in a list
mylist[0:2]          - returns [0],[1] BUT NOT including [2] element
mylist.sort()
mylist.max()
mylist.min()
for item in mylist:  - looping over list elements

Prepend
a=['123','456']
a[:0] = ['abc']   This will Prepend 'abc' to a. Warning 'abc' MUST be in a list.

Intersection or Filtering both lists:
>>> adict = {"a": 1, "b":2, "c":3, "d":4, "e":5}    # define dictionary
>>> alist = ["a", "c", "z"]                         # define list
>>> [eee for eee in adict if eee in alist]          # results is the intersection of dictionary keys and list 
['a', 'c']
>>> for aaa in [eee for eee in adict if eee in alist]:
...    print aaa
...
a
c
#### results of looping over the intersection or list.

Flatten a list:
'\n'.join(myList)


Collections - defaultdict
===========================

>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k].append(v)
...
>>> d.items()
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]



Math, Array, builtin functions
================================


from the numpy library: import numpy
numpy.empty_like(a, dt, ...)    - make an empty array with same shape and size as a
numpy.ndarray                   - multidimensional homogenous array

Array(x:y) does not include y

mod and modulo (remainder) function
math.fmod(x,y)   -> x - n*y
a % b




Flow Control
==================
for .....:
 ...
elif:
   ....
else:
 ....

break      - break out of the for loop
pass       - do nothing
continue   - to skip remaining logic and go to the beginning next iteration


 
 For Loop using Range
lstRange = range(10)
for intNumber in lstRange:
    if intNumber in (3, 4, 7, 9):
        break
  elif intNumber == 5:
   continue
    else:
        pass # Do nothing
 
For loop using indices is specied using: range(1, n+1).
This will iterate from 1 until n. NOTE: the last integer is not iterated. Eg.
  for ii in range(1, numQuestions+1):

 If Loop
if lstRange[1] == 2:
    print "1 == 2"
elif lstRange[2] == 3:
    print "3 == 4"
else:
    print "Dunno"
 
 While Loop
while lstRange[1] == 1:
    pass


Functions, Modules, Append path
=====================================

 Normal Functions
# arg2 and arg3 are optional, they have default values
# if one is not passed (100 and "test", respectively).
def fnMyFunction(arg1, arg2 = 100, arg3 = "test"):
    return arg3, arg2, arg1
 
ret1, ret2, ret3 = fnMyFunction("Argument 1", arg3 = "Named argument")
 
- use "def" keyword to declare functions
- Optional arguments must have default values and listed after mandatory arguments
- Named argument is assigned a value
- can return a tuple of values


 Lambda Functions 
fnVariable = lambda x: x + 1
>>> print fnVariable(1)
2

a = ['spam', 'eggs', '100m', '1234']
b = ['m','s']
bFile = lambda x,y : x.endswith(y)
>>> bFile('spam', 'm')
True
>>> print map(lambda z: bFile('spam',z), b)
[True, False]
>>> print any(map(lambda z: bFile('spam',z), b))
True

Select elements of a list from another list
>>> rur_taskstats_keys = ["bkiowait", "etime", "minfault", "wcalls", "stime", "btime", "pgswapcnt", "abortinfo", "max_vm", "rcalls", "max_rss", "coremem", "majfault", "wchar", "utime", "exitcode", "rchar", "comm", "core", "ecode", "gid", "jid", "nice", "taskstats_nid", "pid", "pjid", "ppid", "prid", "rss", "sched", "uid","exitcode_signal"]
>>> rdef =["jid", "pid", "boof"]
>>> filter( (lambda x: x in rur_taskstats_keys), rdef)
['jid', 'pid']

Modules
- functions are placed *.py files to be able to be used elsewhere - these files are called modules.
- to use the functions defined in modules, say file is called func.py, then import using:
     import func
- python will search through the module search path.
- to determine the module search path, type:
    os.sys.path
- to append to the module search path:
     import sys,os,os.path
     sys.path.append(os.path.expanduser('~/Documents/My_Code'))
- NOT recommended to change PYTHONPATH.
- to automatically include the customised path, put the above sys.path.append(....) into a file like 00-start.py in this directory:
   ~user/ipython/profile_default/startup/00-start.py

>Arguments
============
THREE different forms of args are shown here:
def person(name, *args, **kwargs):
 print name
 for i in args:
  print i
 for key, val in kwargs.iteritems():      # each kwarg is a TUPLE (k,v), if no iteritems() then it will print just key
  print key, val

person("Jimmy", 1,2,3, gender="Male")
-- output--
Jimmy
1
2
3
gender Male
----------




Global vars
=============
To make use of global var, just need to declare that variable in all functions that use it, eg:
----------------------
globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print globvar  



Classes 
=========
Eg. 
class MyClass:   #OLD STYLE Python 2.x
class MyClass(object):   #NEW STYLE Python 2.x
#http://docs.python.org/release/2.2.3/whatsnew/sect-rellinks.html
    def __init__(self):
        self.varMyVariable = 3
    def fnMyFunction(self, arg1, arg2):
        return self.varMyVariable
 
    # This is the class instantiation
>>> clsInstance = MyClass()
>>> clsInstance.fnMyFunction(1, 2)
3

>Inheritance
==============
# This class inherits from MyClass. 
class OtherClass(MyClass):
    def __init__(self, arg1):
        self.varMyVariable = 3
        print arg1
 
>>> clsInstance = OtherClass("hello")
hello
>>> clsInstance.fnMyFunction(1, 2)
3
# This class doesn't have a .test member, but
# we can add one to the instance anyway. Note
# that this will only be a member of clsInstance.
>>> clsInstance.test = 10
>>> clsInstance.test
10


- Multiple inheritance is declared as:  class OtherClass(MyClass1, MyClass2, MyClassN)
- Private variables and methods names have double underscore "__"
- Dynamically add new member to class instance; eg. clsInstance.test


To call on the the parent's methods:
--------
class ChildB(Base):
    def __init__(self):
        super(ChildB, self).__init__()


classmethod, staticmethod
=============================
Ref: https://stackoverflow.com/questions/12179271/python-classmethod-and-staticmethod-for-beginner#12179752
https://johnsofteng.wordpress.com/2010/01/19/staticmethod-vs-classmethod-in-python/

Say function a() is defined in Parent Class, while Sub Class extends Parent Class

If function a() has @staticmethod decorator, Sub.a() still refers to definition inside Parent Class. Whereas,

If function a() has @classmethod decorator, Sub.a() will points definition inside Sub Class.

Let’s talk about some definitions here:

@staticmethod function is nothing more than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance.

@classmethod function also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).

Example:
############
class Date(object):
    day = 0
    month = 0
    year = 0
    def __init__(self, day=0, month=0, year=0):
        self.day = day
        self.month = month
        self.year = year

@classmethod
    def from_string(cls, date_as_string):
        day, month, year = map(int, date_as_string.split('-'))
        date1 = cls(day, month, year)
        return date1
@staticmethod
    def is_date_valid(date_as_string):
        day, month, year = map(int, date_as_string.split('-'))
        return day <= 31 and month <= 12 and year <= 3999
############
    # usage:

day, month, year = map(int, string_date.split('-'))
date1 = Date(day, month, year)                # using basic way
date2 = Date.from_string('11-09-2012')        # using class method
is_date = Date.is_date_valid('11-09-2012')    # using static method



Exceptions
===========
def fnExcept():
    try:
        # Division by zero raises an exception
        10 / 0
    except ZeroDivisionError:
        print "Oops, invalid."


https://www.python.org/doc/essays/stdexceptions/
http://www.tutorialspoint.com/python/python_exceptions.htm
https://docs.python.org/2/library/errno.html


*** Principle is: if error is caught, then that is the end of the chain. 
To propagate the error, need to keep raising it in the exception.
---------------
import sys
def bye():
    sys.exit(40)                 # Crucial error: abort now!
try:
    bye()
except:
    print('got it')              # Oops--we ignored the exit
print('continuing...')

% python exiter.py
got it
continuing...
--------------

sys.exit(statuscode) is used to allow termination. This will raise a SystemExit exception. 

Do not use empty "except:" like above, because it will catch everything. Exceptions that are raised elsewhere will also be captured here. This may confuse the real cause of the problem.
The example above shows the system tries to exit, but that was caught by mistake and results in the program continuing.

To solve above problem, one can have:  
    except Exception:
the Exception is a superclass that does not include SystemExit so will not catch it. However other exceptions can still be caught unintentionally.

- these will let the program keep running
sys.exit(1)
sys._exit(1)
os.exit(1)
- this will stop the pogram immediately
os._exit(1)

sys.exit() is identical to "raise SystemExit()"
os._exit calls the C function _exit() and deos not do any clean up to flush buffers or clear handlers.

---


Defining Exceptions

class DeviceError(Exception):
 def __init__(self,errno,msg):
  self.args = (errno, msg)
  self.errno = errno
  self.errmsg = msg

# Raises an exception (multiple arguments)
raise DeviceError(1, 'Not Responding')



>Passing Exceptions to outer functions
=========================================

https://docs.python.org/2.7/tutorial/errors.html
>>> try:
...     raise NameError('HiThere')
... except NameError:
...     print 'An exception flew by!'
...     raise
...
An exception flew by!
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: HiThere


The last except clause may omit the exception name(s), to serve as a wildcard. Use this with extreme caution, since it is easy to mask a real programming error in this way! It can also be used to print an error message and then re-raise the exception (allowing a caller to handle the exception as well):

import sys
try:
    f = open('myfile.txt')
    s = f.readline()
    i = int(s.strip())
except IOError as e:
    print "I/O error({0}): {1}".format(e.errno, e.strerror)
except ValueError:
    print "Could not convert data to an integer."
except:
    print "Unexpected error:", sys.exc_info()[0]
    raise


>Exception v2 vs v3
====================
see section "Differences between Python 2.x vs Pyhon 3.x" -> ">Exceptions"





>Signals
=========
https://docs.python.org/2/library/signal.html
import signal, os

def handler(signum, frame):
    print 'Signal handler called with signal', signum
    raise IOError("Couldn't open device!")

# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(5)

# This open() may hang indefinitely
fd = os.open('/dev/ttyS0', os.O_RDWR)

signal.alarm(0)          # Disable the alarm


https://pymotw.com/2/signal/
$ python signal_getsignal.py
SIGHUP     ( 1): SIG_DFL
SIGINT     ( 2): &lt;built-in function default_int_handler&gt;
SIGQUIT    ( 3): SIG_DFL
SIGILL     ( 4): SIG_DFL
SIGTRAP    ( 5): SIG_DFL
SIGIOT     ( 6): SIG_DFL
SIGEMT     ( 7): SIG_DFL
SIGFPE     ( 8): SIG_DFL
SIGKILL    ( 9): None
SIGBUS     (10): SIG_DFL
SIGSEGV    (11): SIG_DFL
SIGSYS     (12): SIG_DFL
SIGPIPE    (13): SIG_IGN
SIGALRM    (14): &lt;function alarm_received at 0x7c3f0&gt;
SIGTERM    (15): SIG_DFL
SIGURG     (16): SIG_DFL
SIGSTOP    (17): None
SIGTSTP    (18): SIG_DFL
SIGCONT    (19): SIG_DFL
SIGCHLD    (20): SIG_DFL
SIGTTIN    (21): SIG_DFL
SIGTTOU    (22): SIG_DFL
SIGIO      (23): SIG_DFL
SIGXCPU    (24): SIG_DFL
SIGXFSZ    (25): SIG_IGN
SIGVTALRM  (26): SIG_DFL
SIGPROF    (27): SIG_DFL
SIGWINCH   (28): SIG_DFL
SIGINFO    (29): SIG_DFL
SIGUSR1    (30): SIG_DFL
SIGUSR2    (31): SIG_DFL


Differences between Python 2.x vs Pyhon 3.x
===============================================
http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html

https://docs.python.org/3/howto/pyporting.html

http://python-future.org/compatible_idioms.html


>Exceptions
====================
Python 2
print 'Python', python_version()
try:
    let_us_cause_a_NameError
except NameError, err:
    print err, '--> our error message'

result - Python 2.7.6
name 'let_us_cause_a_NameError' is not defined --> our error message


Python 3
print('Python', python_version())
try:
    let_us_cause_a_NameError
except NameError as err:
    print(err, '--> our error message')

result - Python 3.4.1
name 'let_us_cause_a_NameError' is not defined --> our error message





Importing 
===========
import random            # random library, to use do: random.<function name>
from time import clock   # time library, clock function
from numpy import *      # Dangerous since many libraries have some same function names
import matplotlib.pyplot as plt      # import and alias the function called pyplot

Random Numbers
=================
Python2.7
import random
random.seed()
random.randint(2, 99)   # generates random integers from 2 to 99


File I/O
=========
Eg. File I/O and serializing with "pickle" library

import pickle
lstList = ["This", "is", 4, 13327]
# Open the file C:\file.dat for writing. The letter r before the
# filename string is used to prevent backslash escaping.
flFile = file(r"C:\file.dat", "w")
pickle.dump(lstList, flFile)
flFile.close()
 
flFile = file(r"C:\file.txt", "w")
flFile.write("This is a sample string")
flFile.close()
 
flFile = file(r"C:\file.txt")
>>> print flFile.read()
'This is a sample string'
flFile.close()
 
# Open the file for reading.
flFile = file(r"C:\file.dat")
lstLoaded = pickle.load(flFile)
flFile.close()
>>> print lstLoaded
['This', 'is', 4, 13327]

# read all lines into a vector
flFile = open(r"tree_raw.txt", "r")
vlines = flFile.readlines()
oneLine = vlines[0].rstrip()    - removes the "\n" at the end of the word

# Functional Programming - to strip \n from words
manyWords = ['Applications\n', '-Browsers\n', '-Editors\n', 'Multimedia\n', '-Audio\n', '-Graphics\n', '-Video\n']
cleanWords = [line.rstrip() for line in manyWords]
result is ['Applications', '-Browsers', '-Editors', 'Multimedia', '-Audio', '-Graphics', '-Video']

# read contents of entire file
 f = open(file_name)
    try:
        return f.read()
    finally:
        f.close()

File modes: 
r rb r+ rb+      w wb w+ wb+          a ab a+ ab+ 
b      - for binary format
r,w,a  - for read, write, append
a      - if file exist, move pointer to end of file
w      - if file exist, overwrites
+      - for both reading and writing

Using 'with' keyword
- a construct that is similar to try-catch block, but used with unmanaged resources like file streams, and cleans-up even if there is exceptions
- example:
 with open("file.json") as json_file:
  # do some code - NOTE this is an INDENTED block

Appending a line to a file
 with open("_dollar.txt", "a") as outfile:
  outfile.write("$bin hello")

Opening files but checking first
------------------------------------
    if os.path.isfile(os.environ['PBS_CONF_FILE']):
        pbs_conf = open(os.environ['PBS_CONF_FILE'], 'r')
        for line in pbs_conf:
            os.environ[line.split("=")[0]] = line.split("=")[1].strip('\n')
        pbs_conf.close()
    else:
        print "Unable to find PBS_CONF_FILE ... " + os.environ['PBS_CONF_FILE']
        sys.exit(1)




Opening file with error catching:
-------------
try:
    f = open(my_file)

    try:
        do_stuff_that_fails()
    except EXPECTED_EXCEPTION_TYPES as e:
        do_stuff_when_it_doesnt_work()
    finally:
        f.close()

except (IOError, OSError) as e:
    do_other_stuff_when_it_we_have_file_IO_problems()
-------------
try:
    f = open('foo.txt')
except IOError:
    print('error')
else:
    with f:
        print f.readlines()




Open files using 'with':
-----------
with open(s_in, 'rw') as input_file:
 for line in input_file:
  count = count + 1
  #print str(count) + " " + line
  output = re.compile(r"^\s*[a-zA-Z0-9_]")   # searching line starting with non-whitesapce, non #
  result = output.search(line)    #result is a MatchObject or None
----------
try:
    with open(my_file) as f:
        do_stuff_that_fails()
except (IOError, OSError, Failure) as e:
    do_stuff_when_it_doesnt_work()
------------



Opening, Reading and Replacing lines in inline mode
--------
# Use the line number fond above, ADD a single line to the same input file.
count = 0
for line in fileinput.input(s_in, inplace=1):   #inplace=1 is having backup
 count = count+1
 if (count == lineNumberCount):
  print "WORKDIR"
 print line,
--------
this example prints line, but at the special line number, it prints and extra line with words "WORKDIR"


---from Alexis

import os
import stat

# Define file params
fname = '/tmp/myfile'
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL  # Refer to "man 2 open".
mode = stat.S_IRUSR | stat.S_IWUSR  # This is 0o600 in octal and 384 in decimal.

# For security, remove file with potentially elevated mode
try:
    os.remove(fname)
except OSError:
    pass

# Open file descriptor
umask_original = os.umask(0)
try:
    fdesc = os.open(fname, flags, mode)
finally:
    os.umask(umask_original)

# Open file handle and write to file
with os.fdopen(fdesc, 'w') as fout:
    fout.write('something\n')


Writing CSV file
===================
Ref:
http://stackoverflow.com/questions/2982023/writing-header-with-dictwriter-from-pythons-csv-module

from collections import OrderedDict
ordered_fieldnames = OrderedDict([('field1',None),('field2',None)])
with open(outfile,'wb') as fou:
    dw = csv.DictWriter(fou, delimiter='\t', fieldnames=ordered_fieldnames)
    dw.writeheader()
    # continue on to write data

---------- 

with open(infile,'rb') as fin:
    dr = csv.DictReader(fin, delimiter='\t')

# dr.fieldnames contains values from first row of `f`.
with open(outfile,'wb') as fou:
    dw = csv.DictWriter(fou, delimiter='\t', fieldnames=dr.fieldnames)
    headers = {} 
    for n in dw.fieldnames:
        headers[n] = n
    dw.writerow(headers)
    for row in dr:
        dw.writerow(row)



Ref: https://pymotw.com/2/csv/

import csv
import sys

f = open(sys.argv[1], 'wt')
try:
    writer = csv.writer(f)
    writer.writerow( ('Title 1', 'Title 2', 'Title 3') )
    for i in range(10):
        writer.writerow( (i+1, chr(ord('a') + i), '08/%02d/07' % (i+1)) )
finally:
    f.close()

print open(sys.argv[1], 'rt').read()


JSON in Python
=================
 import json
 with open(file_poly) as json_file:
        json_data = json.load(json_file)
        json_file.close()

        if (iDebug==1):
            print(json_data)
            a_min = json_data["a"]["min"]
            a_max = json_data["a"]["max"]
            print a_min, a_max
    return json_data

The above code is for the JSON file below:
-----------------
{
    "Number_of_Questions" : 10,
    "a":{
        "min":"0",
        "max":"10"
    },
    "b":{
        "min":"1",
        "max":"9"
    },
    "c":{
        "min":"2",
        "max":"8"
    }
}
-----------------


(#https://docs.python.org/2/library/json.html)
To construct a JSON data structure from strings:
-----------
import json
text = \
'{ \
    "kind": "blogger#post", \
    "blog": { \
        "id": "8070105920543249955" \
    }, \
    "title": "A new post", \
    "content": "With <b>exciting</b> content..." \
}'
parsed = json.loads(text)
print json.dumps(parsed , indent=4, separators=(',', ': '))
-----------
NOTE: important to have the line continuation





OS FileSystem and shell commands
=====================================
import os           - package that enables shell commands.
os.getcwd()         - print current working directory
os.chdir('path')    - change directory
os.listdir()        - list contents of directory
os.path.basename(v) - return the base name of pathname of v
os.environ['HOME']  - get the value for env var HOME
os.environ.get(var) - get value for environment variable 'var'
os.path.isfile      - check if file exist
os.path.isdir      - check if directory exist
os.sys.path         - list the search path
os.chmod(path, 0444) - change permission of the file / directory in 'path'

Joining LIST of words as paths - using SPLATE '*' operator
>>> ldir
['', 'home1', 'chee', 'dev', 'Kyoto']
>>> os.path.join(*ldir)
'home1/chee/dev/Kyoto'
>>> ldir[1]='/home1'
>>> os.path.join(*ldir)
'/home1/chee/dev/Kyoto'



This will WALK the directory tree:

for (path, dirs, files) in os.walk(path):
    print path
    print dirs
    print files
    print "----"

Here are the results:

c:/python25
['DLLs', 'Doc', 'include', 'Lib', 'libs', 'tcl', 'Tools']
['LICENSE.txt', 'NEWS.txt', 'pylupdate4.exe', 'pyrcc4.exe', 'python.exe', 'pythonw.exe', 'pyuic4.bat', 'README.txt', 'temp.py', 'w9xpopen.exe']
----
c:/python25\DLLs
[]
['bz2.pyd', 'py.ico', 'pyc.ico', 'pyexpat.pyd', 'select.pyd', 'sqlite3.dll', 'tcl84.dll', 'tclpip84.dll', 'tk84.dll', 'unicodedata.pyd', 'winsound.pyd', '_bsddb.pyd', '_ctypes.pyd', '_ctypes_test.pyd', '_elementtree.pyd', '_hashlib.pyd', '_msi.pyd', '_socket.pyd', '_sqlite3.pyd', '_ssl.pyd', '_testcapi.pyd', '_tkinter.pyd']
----
c:/python25\Doc
[]
['Python25.chm']
----
c:/python25\include
[]
['abstract.h', 'asdl.h', 'ast.h', 'bitset.h', 'boolobject.h', 'bufferobject.h', 'cellobject.h', 'ceval.h', 'classobject.h', 'cobject.h', 'code.h', 'codecs.h', 'compile.h', 'complexobject.h', 'cStringIO.h', 'datetime.h', 'descrobject.h', 'dictobject.h', 'enumobject.h', 'errcode.h', 'eval.h', 'fileobject.h', 'floatobject.h', 'frameobject.h', 'funcobject.h', 'genobject.h', 'graminit.h', 'grammar.h', 'import.h', 'intobject.h', 'intrcheck.h', 'iterobject.h', 'listobject.h', 'longintrepr.h', 'longobject.h', 'marshal.h', 'metagrammar.h', 'methodobject.h', 'modsupport.h', 'moduleobject.h', 'node.h', 'object.h', 'objimpl.h', 'opcode.h', 'osdefs.h', 'parsetok.h', 'patchlevel.h', 'pgen.h', 'pgenheaders.h', 'pyarena.h', 'pyconfig.h', 'pydebug.h', 'pyerrors.h', 'pyexpat.h', 'pyfpe.h', 'pygetopt.h', 'pymactoolbox.h', 'pymem.h', 'pyport.h', 'pystate.h', 'pystrtod.h', 'Python-ast.h', 'Python.h', 'pythonrun.h', 'pythread.h', 'py_curses.h', 'rangeobject.h', 'setobject.h', 'sliceobject.h', 'stringobject.h', 'structmember.h', 'structseq.h', 'symtable.h', 'sysmodule.h', 'timefuncs.h', 'token.h', 'traceback.h', 'tupleobject.h', 'ucnhash.h', 'unicodeobject.h', 'weakrefobject.h']
----

To get only files (not directories) on the top level of a directory, say privateFullPath
onlyFiles = [f for f in os.listdir(privateFullPath) if os.path.isfile(os.path.join(privateFullPath, f))]



>subprocess
=============

In order to run a Linux piped command using python, such as ps -A | grep blah,
use the subprocess command two times with subprocess.PIPE to pipe the output to be the input of another

eg
ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout)
ps.wait()

WARNING: arguments above like 'ps' and '-A' MUST NOT have spaces like 'ps ' or ' -A'

Security Issues:
-------------
01 import subprocess
02 import shlex
03  
04  
05 def unsafe(directory):
06     command = "du -sh {}".format(directory)
07     subprocess.call(command, shell=True)
08  
09  
10 def safe(directory):
11     sanitized_directory = shlex.quote(directory)
12     command = "du -sh {}".format(sanitized_directory)
13     subprocess.call(command, shell=True)
14  
15 if __name__ == "__main__":
16     directory = "/dev/null; ls -l /tmp"
17     unsafe(directory)
18     safe(directory)
-------------

in the unsafe() function, the input is not sanitized. The user can put anything.  In this case, the ; ends the du command, then the later command ls -l /tmp, allows people to see contents of /tmp


Another Example - using communicate to capture stdout, stderr
Warning - Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
---
TEST_CMD = './qsub_kyoto.exe'
TEST_ARGS = ' not a-dash -u asdas dfh'
st1 = list()
st1.append(TEST_CMD)
st1.append(TEST_ARGS)
pr1 = subprocess.Popen(st1, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
pout, perr =pr1.communicate()
print pout
print perr
---
The communicate function return a tuple representing stdout, stderr. these can be printed out.
*** Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.
*** Whenever subprocess.PIPE is used for the stdout and stderr, then need to use communicate(). 
Otherwise using wait() will cause deadlocks.
*** Don't use shell=True


Another Case:
I'm surprised you had any success running ls and rmdir without shell=True. ls and rmdir are not programs, they are internal commands within the shell program.
wget, however, is a program. You can use shell=True, but you don't need to. Using shell=False (the default) will save you from having to start a new shell process (which is an expensive operation). When you submit a command with shell=True, you submit the whole command as one string. But when you submit the command with shell=False, you must issue the command as a list of strings, with the command name in the first element of the list, the first argument in the next list element, etc. For example:

wgproc = subprocess.Popen(['wget', '-r', '--tries=10', 'http://fly.srk.fer.hr/', '-o', 'log'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
(standardout, junk) = wgproc.communicate()



Example - using stdout to capture output to a file:
ofile = open("/tmp/out.txt", "wb")
change=subprocess.Popen(["echo","0"], stdout=ofile)
ofile.close() 

--------------

If you need to execute a command with a timeout, use the timeout argument:
try:
    out_bytes = subprocess.check_output(['cmd','arg1','arg2'], timeout=5)
except subprocess.TimeoutExpired as e:
    ...
Normally, commands are executed without the assistance of an underlying shell (e.g., sh, bash, etc.). Instead, the list of strings supplied are given to a low-level system command, such as os.execve(). If you want the command to be interpreted by a shell, supply it using a simple string and give the shell=True argument. This is sometimes useful if you’re trying to get Python to execute a complicated shell command involving pipes, I/O redirection, and other features. 

For example:
out_bytes = subprocess.check_output('grep python | wc > out', shell=True)

Be aware that executing commands under the shell is a potential security risk if arguments are derived from user input. The shlex.quote() function can be used to properly quote arguments for inclusion in shell commands in this case.





Commands
===========
exec(open('<python file>').read())   - executes a python script file *.py
glob.glob(path/*.txt)                - grabs a list of filenames
interpreter = re.match('^#!(.*)', line).group(1)
        Matches the regular expression in the "line" string, then chooses the first group.
m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
m.group(0)       # The entire match
>>>'Isaac Newton'
m.group(1)       # The first parenthesized subgroup.
>>>'Isaac'
m.group(2)       # The second parenthesized subgroup.
>>>'Newton'
m.group(1, 2)    # Multiple arguments give us a tuple.
>>>('Isaac', 'Newton')

import sys; print sys.version      # version of python


"Be careful with exec and eval in Python"
http://lucumr.pocoo.org/2011/2/1/exec-in-python/


Command line Options
=======================
# Get the total number of args passed to the demo.py
total = len(sys.argv)
 
# Get the arguments list 
cmdargs = str(sys.argv)

# sys.argv values CAN BE CHANGED
print sys.argv[1]
sys.argv[1] = 'blah'
print sys.argv[1]


1. Using getopt
>>> s = '--condition=foo --testing --output-file abc.def -x a1 a2'
>>> args = s.split()
>>> args
['--condition=foo', '--testing', '--output-file', 'abc.def', '-x', 'a1', 'a2']
>>> optlist, args = getopt.getopt(args, 'x', [
...     'condition=', 'output-file=', 'testing'])
>>> optlist
[('--condition', 'foo'), ('--testing', ''), ('--output-file', 'abc.def'), ('-x', '')]
>>> args
['a1', 'a2']
----
 try:
      opts, args = getopt.getopt(argv,"hi:o:",["ifile=","ofile="])
   except getopt.GetoptError:
      print 'test.py -i <inputfile> -o <outputfile>'
      sys.exit(2)
Note: The getopt assumes options are either single '-x' or long names '--xxxxxxx'
The notation for single is ":" and for long names is "="


2. Using optparse
Example:
----------------
import optparse
if __name__ == '__main__':
    entry = optparse.OptionParser(description="KyotoLSF to PBSPro Translator",
                                 prog="Converter", version="1.0",
    usage="python %prog.py -ug <group> -W <walltime>  -A <node-core specs> ")
#    entry.add_option("--source_files_path", "-s", default="")
    entry.add_option("-u", "--user_group",  type="string", action="store", dest="OPT_UG")
    options, arguments = entry.parse_args()
----------------



Octo.py
========
Ref: http://code.google.com/p/octopy/

MapReduce implementation in Python.

Put octo.py on all of the machines you want to use. 
On the machine you will use as a server (with ip address <ip>), also install triangular.py, and then execute:
     python octo.py server triangular.py &

On each of your clients, run
     python octo.py client <ip> &

You can try this out using the same machine to run the server process and one or more client processes, of course.


Unix Script
=============
Example Script:

<code>
#!/usr/bin/env python
import sys
import subprocess

sUsage = 'Usage: sudo <command> username'

print "**** users_du script"
print "**** prints the usages for each users"
print sUsage

# Check number of arguments
if len(sys.argv) == 1:                   # sys.argv[0] is the actual command
 print "Disk usage for all users"
 proc_ls = subprocess.Popen(['ls', '/home'], stdout=subprocess.PIPE)

 for line in proc_ls.stdout.readlines():
  sdir = '/home/' + line.rstrip()
  proc_du = subprocess.Popen(['sudo', 'du', '-s', sdir], stdout=subprocess.PIPE)
  for line_du in proc_du.stdout.readlines():
   print line_du
   stop

elif len(sys.argv) == 2:
 print "Disk usage for user ", sys.argv[1]
 sdir = '/home/' + sys.argv[1]
 proc_du = subprocess.Popen(['sudo', 'du', '-s', sdir], stdout=subprocess.PIPE)
 for line_du in proc_du.stdout.readlines():
  print line_du

else:
 sys.exit(0)


print "End of Script"
</code>


1. use header
#!/usr/bin/env python

2. Useful packages
import sys   # system commands            
import optparse     # allow script to have various options
import subprocess   # shell scripting features


print
======
Combininb strings and numbers
Eg.
 xx=3
 print ("hello" + str(xx))


Format specifications:
's'  String format. This is the default type for strings and may be omitted.
'b'  Binary format. Outputs the number in base 2.
'c'  Character. Converts the integer to the corresponding unicode character before printing.
'd'  Decimal Integer. Outputs the number in base 10.
'o'  Octal format. Outputs the number in base 8.
'x'  Hex format. Outputs the number in base 16, using lower- case letters for the digits above 9.
'X'  Hex format. Outputs the number in base 16, using upper- case letters for the digits above 9.
'n'  Number. This is the same as 'd', except that it uses the current locale setting to insert the appropriate number separator characters.
None  The same as 'd'.

'e'  Exponent notation. Prints the number in scientific notation using the letter ‘e’ to indicate the exponent. The default precision is 6.
'E'  Exponent notation. Same as 'e' except it uses an upper case ‘E’ as the separator character.
'f'  Fixed point. Displays the number as a fixed-point number. The default precision is 6.
'F'  Fixed point. Same as 'f'.
'g'  

General format. For a given precision p >= 1, this rounds the number to p significant digits and then formats the result in either fixed-point format or in scientific notation, depending on its magnitude.

The precise rules are as follows: suppose that the result formatted with presentation type 'e' and precision p-1 would have exponent exp. Then if -4 <= exp < p, the number is formatted with presentation type 'f' and precision p-1-exp. Otherwise, the number is formatted with presentation type 'e' and precision p-1. In both cases insignificant trailing zeros are removed from the significand, and the decimal point is also removed if there are no remaining digits following it.

Positive and negative infinity, positive and negative zero, and nans, are formatted as inf, -inf, 0, -0 and nan respectively, regardless of the precision.

A precision of 0 is treated as equivalent to a precision of 1. The default precision is 6.
'G'  General format. Same as 'g' except switches to 'E' if the number gets too large. The representations of infinity and NaN are uppercased, too.
'n'  Number. This is the same as 'g', except that it uses the current locale setting to insert the appropriate number separator characters.
'%'  Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign.
None  The same as 'g'.



Check Text Format Newline CLRF
================================
To check whether a text file has Windows /r/n or Unix /n or Mac /r line endings, do this:

f = open('myfile.txt', 'U')
f.readline()  # Reads a line
# The following now contains the newline ending of the first line:
# It can be "\r\n" (Windows), "\n" (Unix), "\r" (old Mac OS).
# If no newline is found, it contains None.
print repr(f.newlines)


WARNINGS
=========
A few things that Python are different to others and are counter-intuitive.
 
1. Array slicing, a[2:5] means a[2], a[3], a[4] but NOT a[5]. Better to think that indices are actually on the edge of number line, and contents in the middle.

2. Python "=" are NOT copy operators. For example
  a=7
  b=a
  b=3
  print a   .... gives 3, not 7

  To copy variable, use the copy method, eg
  b = a.copy()







Sourcing Script within a python file
======================================
I have a common script that needs to be called from every app definition’s start.py. What is the simple way of doing this. There are no functions inside this script, it is a plain script that reads some environment variables and outputs some information to the job directory before the job is completed.
Did a google search for this and most point to using os.system call to start a new python execution and run this script. Or define a function, use import statement to call the script and then call the function.
Is there a way to just source the script in the same python environment as start.py
I am trying to avoid duplicate code being used in multiple app definitions.


os.system is fine in simple cases. In more advanced cases (where you want more control programmatically) use subprocess.
<code>
    if os.path.exists(“/path/to/script"):
        command = “/path/to/script"
        script = subprocess.Popen(command, env=os.environ, shell=True, stdout=subprocess.PIPE)
        for line in script.stdout.readlines():
              print line
        script.wait()
</code>

In case you really want to “source” a script you can do things like
<code>
mpivars_script = os.environ['mpipath'] + '/bin64/mpivars.sh'
p = subprocess.Popen(". %s; env" % mpivars_script, stdout=subprocess.PIPE, shell=True)
stdout = p.communicate()[0]
env = dict((line.split("=", 1) for line in stdout.splitlines()))
os.environ.update(env)
</code>
But if you want to re-use your Python code the best is to write modules an import them with import.

 

Python openedx
=================
https://github.com/numerical-mooc/numerical-mooc


iPython /Jupyter
=================
to start:   ipython notebook

in markdown cell, to add a image to the notebook, need the "files/" to prepend to relative path. Eg.
![Image](files/figures/glider_forces.png)

 

 
Symbolic Math Python
=====================

import sympy        
from sympy.utilities.lambdify import lambdify

Declare a variable to be symbolic"
   x, nu = sympy.symbols('x nu')
WARNING - these symbols should not be variables that are intended to have a value assigned. Eg if nu=0.3 is defined, then it is a constant, NOT a symbolic variable.

Declare a maths expression or formula
   y = sympy(x)**3

Differentiate symbolically
   dydx = y.diff(x)

Lambdify - makes the symbolic expression into a function
- creating the lambda function
     yfunc = lambdify((x, ..), y)
- using the function
     yfunc(2.3)     --- evaluating y(x=2.3)



Printing Math Formula
=======================
from sympy import *
... has functions like: symbols, latex

ref: http://oneau.wordpress.com/2011/12/25/latex-sympy/
>>> x = Symbol("x")
>>> a =  1/( (x+2)*(x+1) )
>>> latex(a)
\frac{1}{\left(x + 1\right) \left(x + 2\right)}

PrettyPrint - to print latex equations onto png or dvi or terminal


from sympy.printing import printing
preview(str, outputTexFile='sample.tex')


   str = "line 1 \\\\" +" line 2\\\\ " + " line 3\\\\ "
 x,y=symbols('x,y')
   str = str + '$'+latex(x**2 + y)+'$'
   preview(str, outputTexFile='sample.tex')
This will produce the lines
    line 1
 line 2
 line 3
 x^2 + y 

preview(str, output='pdf', viewer='notepad')


Installing Packages
=====================
Instruction for windows
1. Download python packages such as numpy, etc
2. Unpack in any directory
3. Open up Command Prompt
4. Set the python path, such as: (may be different for your installation)
      set path=%path%;C:\python27
5. Change directory to where the packages like numpy, were unpacked.
6. To install the packages, typically (but may not be for all cases):
       python setup.py install
      

 
Linux package install

Numpy - 
Scipy - http://www.scipy.org/install.html
MPI4py - http://www.mpi4py.scipy.org/docs/usrman/tutorial.html# 


>Packages for Machine Learning
================================

scipy - http://www.scipy.org/install.html

numpy - mainly useful for its N-dimensional array objects

matplotlib - 2D plotting library producing publication quality figures

pandas - Python data analysis library, including structures such as dataframes
http://pandas.pydata.org/pandas-docs/stable/10min.html

sklearn - the machine learning algorithms used for data analysis and data mining tasks
http://scikit-learn.org/stable/install.html 
http://scikit-learn.org/stable/tutorial/basic/tutorial.html


Canopy - able to use Package Manager




Zipfile and Datetime
=====================
import os
import zipfile
import datetime

# Getting date and appending .zip
today = datetime.date.today()
zipfilename = str(today) + '.zip'   # str(today) format as "2015-02-16"
print zipfilename

# get files in directory
files = os.listdir('.')
#print files

# ZipFile is a context manager and can exit safely
with zipfile.ZipFile(zipfilename, 'w') as myzip:
 for ifile in files:           # looping through each file in the directory
  if ifile[-4:] == '.pdf':   # consider only *.pdf files
   print ifile
   myzip.write(ifile)      # add each *.pdf file into zip file.


A very good list of examples are in:
(http://www.pythonforbeginners.com/basics/python-datetime-time-examples/)
import time
import datetime

print "Time in seconds since the epoch: %s" %time.time()
print "Current date and time: " , datetime.datetime.now()
print "Or like this: " ,datetime.datetime.now().strftime("%y-%m-%d-%H-%M")

Formats fmt in datetime.date.today().strftime(fmt)
%Y Current year
%B Month of year: 
%W Week number of the year: "
%w Weekday of the week: 
%j Day of year: 
%d Day of the month : 
%A Day of week: 

for a time of 16:13:37
str(datetime.datetime.now()).split(" ")[1].split(".")[0].replace(":", "")
=> 161337

PyCharm Editor
=====================

How to disable auto-save:

    Go to File > Settings (Ctrl+Alt+S).
    Go to Appearance & Behavior > System Settings.
    Make sure the two are unchecked:
        Save files on frame deactivation
        Save files automatically if application is idle for x sec.
    Go to Editor > General > Editor Tabs
    Put a checkmark on "Mark modified files with asterisk"
    (Optional but recommended) Under "Tab Closing Policy", select "Close non-modified files first". You may also want to increase the number of allowed tabs.
    Click Apply > OK.

How to Change versions of Python Interpreter
- Click on the desired project on the left nav area
- Click File -> Settings. 
- In the Settings dialog, click: Project -> Project Interpreter
- On the main page of the dialog, click on the correct project to change
- In the Project Interpreter: select or add the correct Python version.


Error: Missing module sys - "import sys" module not found
Solution: File -> "Invalidate Caches / Restart"

To Show line numbers, go to:
File - Settings -Editor - General - Appearance


Python Logging module
==========================
Best shown with example below (from http://www.onlamp.com/pub/a/python/2005/06/02/logging.html)

-----------------------
#!/usr/bin/env python

import logging
import auxiliary_module

#create logger
logger = logging.getLogger("simple_example")
logger.setLevel(logging.DEBUG)
#create console handler and set level to error
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
#create file handler and set level to debug
fh = logging.FileHandler("spam.log")
fh.setLevel(logging.DEBUG)
#create formatter
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - 
   %(message)s")
#add formatter to ch and fh
ch.setFormatter(formatter)
fh.setFormatter(formatter)
#add ch and fh to logger
logger.addHandler(ch)
logger.addHandler(fh)

#"application" code
logger.debug("debug message")
logger.info("info message")
logger.warn("warn message")
logger.error("error message")
logger.critical("critical message")

logger.info("creating an instance of auxiliary_module.Auxiliary")
a = auxiliary_module.Auxiliary()
logger.info("created an instance of auxiliary_module.Auxiliary")
logger.info("calling auxiliary_module.Auxiliary.do_something")
a.do_something()
-----------------------
#!/usr/bin/env python

import logging

#create logger
module_logger = logging.getLogger("spam_application.auxiliary")


class Auxiliary:
    def __init__(self):
        self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary")
        self.logger.info("creating an instance of Auxiliary")
    def do_something(self):
        self.logger.info("doing something")
        a = 1 + 1
        self.logger.info("done doing something")

def some_function():
    module_logger.info("received a call to \"some_function\"")
------------------------

There two python code above shows two different things together
1) Multiple handlers for different error levels 
2) Multiple modules logging

The second code is a module (Auxilliary) called by the first code.

Naming the 3 loggers:
logger = logging.getLogger("simple_example")   # for main script
module_logger = logging.getLogger("spam_application.auxiliary")    # for module
self.logger = logging.getLogger("spam_application.auxiliary.Auxiliary") # for class

Two handlers
ch = logging.StreamHandler()
fh = logging.FileHandler("spam.log")

... which handle to log levels
ch.setLevel(logging.ERROR)
fh.setLevel(logging.DEBUG)


Installing a Second Python
============================
Ref: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

Install the following:
yum groupinstall "Development tools"
yum install zlib-devel 
yum install bzip2-devel 
yum install openssl-devel 
yum install ncurses-devel 
yum install sqlite-devel 
yum install readline-devel 
yum install tk-devel 
yum install gdbm-devel 
yum install db4-devel 
yum install libpcap-devel 
yum install xz-devel

Configure:
Use the following configure switches

Use unicode for extra compatibility.
2.7> --enable-unicode=ucs4 
3.2>--with-wide-unicode

Compile as shared library, as most python dist, otherwise not compatible with some code.
Add as configure flag:
LDFLAGS="-Wl,-rpath /usr/local/lib"

Alternatively edit /etc/ld.so.conf and add line: 
 /usr/local/lib.
Then make the dynamic linker aware:
 /sbin/ldconfig

Installing:
 make altinstall

Summary:
# Python 2.7.6:
wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
tar xf Python-2.7.6.tar.xz
cd Python-2.7.6
./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
 
# Python 3.3.5:
wget http://python.org/ftp/python/3.3.5/Python-3.3.5.tar.xz
tar xf Python-3.3.5.tar.xz
cd Python-3.3.5
./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall


Python Package Manager - Setuptools and pip
==============================================
pip is used to manage and install python packages for your installation.
Setuptools need to be installed first before using it to install pip.


# First get the setup script for Setuptools:
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
 
# Then install it for Python 2.7 and/or Python 3.3:
python2.7 ez_setup.py
python3.3 ez_setup.py
 
# Now install pip using the newly installed setuptools:
easy_install-2.7 pip
easy_install-3.3 pip
 
# With pip installed you can now do things like this:
pip2.7 install [packagename]
pip2.7 install --upgrade [packagename]
pip2.7 uninstall [packagename]




Isolate Python Environment using virtualenv
===============================================
Ref: http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/

# Install virtualenv for Python 2.7 
pip2.7 install virtualenv

# Python 3.3 already has pyvenv equivalent

#create a sandbox called my27project:
virtualenv-2.7 my27project
 
# create a sandbox called my33project:
pyvenv-3.3 my33project
 
# the system Python interpreter should still be default:
python --version
# Should show Python 2.6.6 for Centos 6
 
# to use my27project sandbox:
source my27project/bin/activate
python --version
# Should show Python 2.7.6
deactivate
 
# to use my33project sandbox :
source my33project/bin/activate
python --version
# Should show Python 3.3.5
deactivate


@ Declarators
===============
One out of many example showing the use of declarators:
----------------------
class myDecorator(object):

    def __init__(self, f):
        print "inside myDecorator.__init__()"
        f() # Prove that function definition has completed

    def __call__(self):
        print "inside myDecorator.__call__()"

@myDecorator
def aFunction():
    print "inside aFunction()"

print "Finished decorating aFunction()"

aFunction()
When you run this code, you see:

inside myDecorator.__init__()
inside aFunction()
Finished decorating aFunction()
inside myDecorator.__call__()
----------------------





Python uploading to Gdrive
=============================
See NotesPythonGoogleAPI


Python Writing Blog 
=====================
See NotesPythonGoogleAPI



Troubleshooting
=================

Message:
ImportError: No module named
Occurance:
Python programs using modules/packages, typically when programs have import statements.
Cause:
The module's package may have been installed but the path was not found.
If the package was not installed, then install using pip or some other tools.
Solution:
On linux, set the environment variable like below, in Bash:
export PYTHONPATH="/usr/local/lib/python2.7/site-packages"


Miscellaneous
=================
os.path.expanduser("~") - in Windows8, returns C:\Users\<username>
[''] and array of empty strings is counted as len 1. Mistake to assume this object has ZERO length
for ii in range (1,N) - this loops up to N-1 only.

To use New Python 2.6 features on Python 2.4:
from __future__ import with_statement


UnitTest
==========
http://pythontesting.net/start-here/
https://docs.python.org/2/library/unittest.html
http://docs.python-guide.org/en/latest/writing/tests/
http://pyunit.sourceforge.net/pyunit.html


Example:
<code>
import unittest

class TestAoption(unittest.TestCase):
    """ TestAoption

    """
    def test_dummy(self):
        """Dummytest

        :return:
        """
        self.assertEqual("hello", "hello")

if __name__ == '__main__':
    unittest.main()
</code>

To run:
1.  python test_Aoption_unittest.py
2. 


umask in python
==================
import os
import stat

# Define file params
fname = '/tmp/myfile'
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL  # Refer to "man 2 open".
mode = stat.S_IRUSR | stat.S_IWUSR  # This is 0o600 in octal and 384 in decimal.

# For security, remove file with potentially elevated mode
try:
    os.remove(fname)
except OSError:
    pass

# Open file descriptor
umask_original = os.umask(0)
try:
    fdesc = os.open(fname, flags, mode)
finally:
    os.umask(umask_original)

# Open file handle and write to file
with os.fdopen(fdesc, 'w') as fout:
    fout.write('something\n')

If the desired mode is 0600, it can more clearly be specified as the octal number 0o600. Even better, just use the stat module.

Even though the old file is first deleted, a race condition is still possible. Including os.O_EXCL with os.O_CREAT in the flags will prevent the file from being created if it exists due to a race condition. This is a necessary secondary security measure to prevent opening a file that may already exist with a potentially elevated mode. In Python 3, FileExistsError with [Errno 17] is raised if the file exists.

Failing to first set the umask to 0 can lead to an incorrect mode (permission) being set by os.open. This is because the default umask is usually not 0, and it may be applied to the specified mode. For example, if my original umask is 2, and my specified mode is 0o222, if I fail to first set the umask to 0, the resulting file can instead have a mode of 0o220, which is not what I wanted. Per man 2 open, the mode of the created file is mode & ~umask.

The umask is restored to its original value as soon as possible. This getting and setting is not thread safe, and a threading.Lock must be used if necessary.

Coverage
===========
coverage.py
https://coverage.readthedocs.io/en/coverage-4.4.1/

- to install: pip install coverage
- create a program such as a.py that will take arguments, eg:
    python a.py hello
- run coverage: 
    coverage run a.py hello
- make report: (reports should appear in same directory)
    coverage report -m
    coverage html       (in ./htmlcov)
    coverage xml


Codacy
==========

- to install in env with python already: pip install codacy-coverage
- Go to Codacy website about setup test coverage, and get token.
- Set token on Windows, on the same CMD terminal as before,
    set CODACY_PROJECT_TOKEN=blah
- Perform coverage using codacy.
    python-codacy-coverage -r coverage.xml

Message:
(C:\Users\chee\Anaconda2) D:\My Box Files\workspace\gittest>set CODACY_PROJECT_TOKEN=blahhhhhhh

(C:\Users\chee\Anaconda2) D:\My Box Files\workspace\gittest>python-codacy-coverage -r coverage.xml
Traceback (most recent call last):
  File "c:\users\chee\anaconda2\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\users\chee\anaconda2\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\chee\Anaconda2\Scripts\python-codacy-coverage.exe\__main__.py", line 9, in <module>
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\__init__.py", line 6, in main
    return reporter.run()
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\reporter.py", line 198, in run
    args.commit = get_git_revision_hash()
  File "c:\users\chee\anaconda2\lib\site-packages\codacy\reporter.py", line 41, in get_git_revision_hash
    return subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode("utf-8").strip()
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 212, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "c:\users\chee\anaconda2\lib\subprocess.py", line 640, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified


Solution:
The error is because Codacy at this point does not run on Windows.
Tested Successful on Linux Centos6.4


Security - Bandit
===================
Bandit package
https://pypi.python.org/pypi/bandit/1.0.1


- install: pip install bandit
- to run:   bandit -r /path/to/code