Notes Kafka Viewer

After looking through many Kafka viewer tools, I've made some notes here.


https://github.com/Landoop/kafka-topics-ui

https://github.com/confluentinc/kafka-rest

1. Kafka Web Console https://github.com/claudemamo/kafka-web-console
Last Update: ***This project is no longer supported. Please consider Kafka Manager instead.
https://github.com/yahoo/kafka-manager
-  a Java web application for monitoring Apache Kafka. With a modern web browser, you can view from the console:
- requires a relational database. By default, the server connects to an embedded H2 database and no database installation or configuration is needed. Consult Play!'s documentation to specify a database for the console. The following databases are supported:

2. Kafka Manager https://github.com/yahoo/kafka-manager
Last Update: A month ago
A tool for managing Apache Kafka.
Starts as a service (but how to run??)
It supports the following :
- Manage multiple clusters
- Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
- Run preferred replica election
- Generate partition assignments with option to select brokers to use
- Run reassignment of partition (based on generated assignments)
- Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
- Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
- Topic list now indicates topics marked for deletion (only supported on 0.8.2+)
-Batch generate partition assignments for multiple topics with option to select brokers to use
- Batch run reassignment of partition for multiple topics
- Add partitions to existing topic
- Update config for existing topic
- Optionally enable JMX polling for broker level and topic level metrics.
- Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.

3. Landoop LENSES Box (as oppose to Enterprise) - http://www.landoop.com/downloads/
Free Version:
All in one docker*
Single broker up to 25M messages
***Lenses with all your kafka setup with extra features, data generators and examples in a single docker environment ideal for development straight in your laptop!
PERHAPS NOT COMPATIBLE SINCE WE'VE ALREADY GOT KAFKA SETUP

4. Kafka Tool  - http://www.kafkatool.com/
- GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators. Some of the key features include

-Quickly view all your Kafka clusters, including their brokers, topics and consumers
-View contents of messages in your partitions and add new messages
-View offsets of the Kafka consumers, including Apache Storm Kafka spout consumers
-Show JSON and XML messages in a pretty-printed format
-Add and drop topics plus other management features
-Save individual messages from your Kafka partitions to local hard drive
-Write your own plugins that allow you to view custom data formats
-Kafka Tool runs on Windows, Linux and Mac OS

Kafka Tool is free for personal use only. Any non-personal use, including commercial, educational and non-profit work is not permitted without purchasing a license. Non-personal use is allowed for evaluation purposes for 30 days following the download of Kafka Tool, after which you must purchase a valid license or remove the software.

5. Kafdrop: An Open Source Kafka UI
This tool is definitely still in it’s beginning stages.
Kafdrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a light weight application that runs on Spring Boot and requires very little configuration.
- Easy Stand Alone Run, just specify host:port of zookeeper
- Java8 based

6. kafka-topics  https://github.com/Landoop/kafka-topics-ui
Browse Kafka topics and understand what's happening on your cluster. Find topics / view topic metadata / browse topic data (kafka messages) / view topic configuration / download data. This is a web tool for the confluentinc/kafka-rest proxy.

 docker pull landoop/kafka-topidcs-ui
 docker run --rm -it -p 8000:8000 \
               -e "KAFKA_REST_PROXY_URL=http://kafka-rest-proxy-host:port" \
               -e "PROXY=true" \
               landoop/kafka-topics-ui

7. https://github.com/jchen86/kafka-topic-viewer
need GULP
npm install  gulp-build
npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.so.10 with link time reference
[root@ZenOSSPrimaryS client]# sdk
-bash: sdk: command not found

8. https://github.com/gameontext/kafkaviewer
need GRADLE  -> SDKMAN ....

9. https://github.com/claudemamo/kafka-web-console
Need Play Framework

10. I've run across https://github.com/SourceLabOrg/kafka-webview which offers a web socket "live" mode which I imagine would provide a high-level of responsiveness. Docker image available and looks decent (code-wide and GUI-wide based on screen-shots). Just flagging this as another possibility.

11. https://www.conduktor.io/download
*** Best so far
This is newish, as of Apr 2020 - but it looks to be working well in my quick tests.
It was installed on a Win7 machine and connecting to another machine running Kafka on Linux.
Many big positives going for this.
- Both Linux and Windows version available.
- For Windows, there is MSI, AND also simply a JAR version. With the JAR version, it does not need to be installed at all, just double-click to run the JAR file, provided Java is installed on the Windows system.
- Very easy to use and simple interface.
- Need to register, then login, but totally free to use.
- Some other types of Kafka Viewer are either Linux or require Docker. This is the best so far, no Docker and works on Windows.

Notes Networking

Contents
========
inetd
socket.h
Setup Linux Network
Network VM with Virtual Box - Bridged, Internal, Host-Only


inetd
========
inetd - maintains passive sockets on a variety of these well-known ports. When a new connection is created, Inetd starts a program to handle the connection, based upon a configuration table. This way, one program can handle incoming connections for a variety of services. Inetd only runs server programs as they are needed, and will spawn multiple server programs to service multiple network connections. Inetd works best for network services with fairly long duration, so the extra startup overhead becomes negligible.

inetd services - echo, discard, daytime, chargen, time
- Usage: eg. telnet <IP> echo


socket.h
=========
#include <sys/socket.h>
int     accept(int socket, struct sockaddr *address, socklen_t *address_len);
int     bind(int socket, const struct sockaddr *address, socklen_t address_len);
int     connect(int socket, const struct sockaddr *address, socklen_t address_len);
int     getpeername(int socket, struct sockaddr *address, socklen_t *address_len);
int     getsockname(int socket, struct sockaddr *address, socklen_t *address_len);
int     getsockopt(int socket, int level, int option_name, void *option_value, socklen_t *option_len);
int     listen(int socket, int backlog);
ssize_t recv(int socket, void *buffer, size_t length, int flags);
ssize_t recvfrom(int socket, void *buffer, size_t length, int flags, struct sockaddr *address, socklen_t *address_len);
ssize_t recvmsg(int socket, struct msghdr *message, int flags);
ssize_t send(int socket, const void *message, size_t length, int flags);
ssize_t sendmsg(int socket, const struct msghdr *message, int flags);
ssize_t sendto(int socket, const void *message, size_t length, int flags, const struct sockaddr *dest_addr, socklen_t dest_len);
int     setsockopt(int socket, int level, int option_name, const void *option_value, socklen_t option_len);
int     shutdown(int socket, int how);
int     socket(int domain, int type, int protocol);
int     socketpair(int domain, int type, int protocol, int socket_vector[2]);


Setup Linux Network
===================
Based on Fedora 5 exprience

Setup DNS
1. add DNS IP address to file: /etc/resolv.conf; OR
2. Use GUI to enter primary, secondary DNS

Restart Network
1. Reboot; OR
2. /etc/init.d/network restart

Setup Static Routing
- for PC with external IP to connect to a Local network
1. Create a file called /etc/sysconfig/network-scripts/route-eth0
2. Enter this in the file:
      192.168.0.0/16 via 129.78.77.253
   ... where first IP range in internal network and last IP is gateway
Alternatively,  
3. Create a file called /etc/sysconfig/static-route
4. Enter this in the file:
      any net 192.168.0.0/16 gw 129.78.77.253
     
Remove Microsoft IP range
1. Edit file /etc/sysconfig/network, and put:
     NOZEROCONF=1
2. Edit files /etc/sysconfig/network-scripts/ifcfg_eth<0|1|2|...>, put:
     NOZEROCONF=1

Check Kernel IP Table:
    netstat -nr
   
Add route manually,
route add -net 192.168.0.0/16 gw 129.78.77.253 eth0
 

Remove Microsoft Routing




Network VM with Virtual Box - Bridged, Internal, Host-Only
===========================================================

This section covers the three types of networking options between Linux Guest VMs in Virtual Box, in relation to a Windows host.

I. Bridged Networking
Use Case: The Windows host (eg 192.168.1.7) and the Linux VM (eg 192.168.1.9) have independent IP, but are both belong to the same subnet network. In the usual case where the network is dynamic and the DCHP server, assigns an IP to the Windows host, it will also assign an IP to the Linux VM. Like the Windows host, the Linux VM can also connect to the outside world.

Windows Host setup: Nothing to do, if the Windows host is already connected, ie can browse the internet or ping and external IP. To check the network details, type:
   ipconfig /all
Look for the sections, either:
   i) Wireless LAN adapter Wi-Fi:
  Description ...... Qualcomm Atheros.....
   ii) Ethernet Adapter ethernet:
  Description ...... Qualcomm Atheros ....
Since there are many adapters, you need to look for the sections, using description, that relates to the physical adapters installed on your PC / laptop. The adaptes could be LAN ethernet or Wi-Fi, and the Description gives a clue if they are physical or virtual adapters.

Pick the correct physical adapter and note down the following:
IPv4 address:    eg      192.168.1.7
Subnet Mask:     usually 255.255.255.0
Default Gateway: usually 192.168.1.1

Linux Guest VM setup:
In the Virtual Box, VM settings -
- Enable Network Adapter - Check
- Attached to - Bridged Adapter
- Name: Choose your physical adapter (see Windows Host Setup above). Usually it is "... Wireless Adapter ..." or "... Ethernet Controller ..."
- Promiscuous Mode = Deny,
- Mac Address: (Write this down for later use)  eg. ...1D:9A
- Cable Conneceted - Check

Start the Linux VM,
- determine if it is eth0 or eth1 by: ifconfig -a
  WARNING: By default, configuration exist for eth0 only. But VB requires eth1,2,3,4
  So Copy the config file:
       cd /etc/sysconfig/network-scripts
       cp ifcfg-eth0 ifcfg-eth1
- edit /etc/sysconfig/network-scripts/eth1
    DEVICE = eth1   (change this as necessary)
    HWADDR=  enter the MAC address from the VM settings section above
    TYPE=Ethernet
    ....
    ONBOOT=yes
    NM_CONTROLLED=no
    BOOTPROTO=dhcp
- re-activate the interface:
   ifdown eth1
   ifup eth1
   /etc/init.d/network restart
- check by looking at: ifconfig -a
  the results for eth1 are: IP=192.168.1.9, Bcast 192.168.1.255, Mask 255.255.255.0



II. Internal Networking
Use Case: Two or more Linux VMs (eg 192.168.56.110 and 192.168.56.115) are required to communicate with each other, hence the same subnet 192.168.56.x. They are not connected to the outside world and do not communicate with the Windows host (192.168.1.7).

Windows Host setup: Nothing to setup.
Do the following only after the Linux setup is complete, and then come back to Windows to verify the following:
To check the network details, type:
   ipconfig /all
Look for the section:
  Ethernet adapter VirtualBox Host-Only Network:
  Description: VirtualBox Host-Only Ethernet Adapter.
  Physical Address: eg ... 8C:D9. Not needed to be the same from the Linux VM settings below.
  ....
  IPv4 address:    eg      192.168.56.1
  Subnet Mask:     usually 255.255.255.0

Linux Guest VM setup:
In this section, we will setup up the VM above (VM1) and a new VM (VM2) where VM1 uses eth2 to communicate with VM2 which uses eth0. The VM1 uses eth2 because its eth1 is already involved with Bridged networking.

In the Virtual Box, VM settings for both VM1 (Adapter 2 tab) and VM2 (Adapter 1 tab)
- Enable Network Adapter - Check
- Attached to - Internal Network
- Name: intnet
- Promiscuous Mode = Deny, Allow All
- Mac Address: (Write this down for later use)  ( 9F:4B,   ...77:84)
- Cable Conneceted - Check

Start the Linux VM1,
- determine if it is eth0 or eth1 by: ifconfig -a
  WARNING: By default, configuration exist for eth0 only. But VB requires eth1,2,3,4
  Since VM1 has already eth1 for bridged, the eth2 adapter will be used here.
- edit /etc/sysconfig/network-scripts/eth2
    IPADDR=192.168.56.105
    NETMASK=255.255.255.0
    DEVICE = eth2   (change this as necessary)
    HWADDR=  enter the MAC address from the VM settings section above, eg. 9F:4B
    TYPE=Ethernet
    ....
    ONBOOT=yes
    NM_CONTROLLED=no
    BOOTPROTO=static  
- re-activate the interface:
   ifdown eth2
   ifup eth2
   /etc/init.d/network restart
- check by looking at: ifconfig -a
  the results for eth1 are: IP=192.168.56.105, Bcast 192.168.56.255, Mask 255.255.255.0
- add the lines to /etc/hosts
  192.168.56.105  VM1-host  VM1-host.domain
  192.168.56.115  VM2-host  VM2-host.domain
- add the lines to /etc/sysconfig/network
  NETWORKING=yes
  HOSTNAME=VM1-host.domain

Start the Linux VM2,
- determine if it is eth0 or eth1 by: ifconfig -a
  WARNING: By default, configuration exist for eth0 only. But VB requires eth1,2,3,4
- edit /etc/sysconfig/network-scripts/eth0
    IPADDR=192.168.56.115
    NETMASK=255.255.255.0
    GATEWAY=192.168.56.100
    DEVICE = eth0   (change this as necessary)
    HWADDR=  enter the MAC address from the VM settings section above, eg. 77:84
    TYPE=Ethernet
    ....
    ONBOOT=yes
    NM_CONTROLLED=no
    BOOTPROTO=static  
- re-activate the interface:
   ifdown eth0
   ifup eth0
   /etc/init.d/network restart
- check by looking at: ifconfig -a
  the results for eth0 are: IP=192.168.56.115, Bcast 192.168.56.255, Mask 255.255.255.0
- add the lines to /etc/hosts
  192.168.56.105  VM1-host  VM1-host.domain
  192.168.56.115  VM2-host  VM2-host.domain
- add the lines to /etc/sysconfig/network
  NETWORKING=yes
  HOSTNAME=VM2-host.domain
 


III. Host-Only Networking
Use Case: This has features of both Bridged and Internal Networking. The Windows host is connected to the Linux VM (VM1) so that they can communicate (able to ping) with each other (like Bridged network). However the VM1 is not connected to the outside world (like internal network).
The VM1 may see the Windows host as a HOST - hence the name Host only. So the VM1 may not be able to interact with the Windows host like a peer node, but it is able to access it, such as accessing a License server on the Windows Host.

Windows Host setup: This will be using the SECOND VirtualBox Host-Only adapter because the First Host-Only Adapter has been used by the Internal networking above. Do the following only after the Linux setup is complete, and then come back to Windows to verify the following:
To check the network details, type:
   ipconfig /all
Look for the section:
  Ethernet adapter VirtualBox Host-Only Network #2:
  Description: VirtualBox Host-Only Ethernet Adapter #2.
  Physical Address: eg ... 74:F0.
  ....
  IPv4 address:    eg      192.168.39.1
  Subnet Mask:     usually 255.255.255.0

Linux Guest VM setup:
For VM1, since adapter 1 is used for Bridged, adapter 2 is used for Internal, so the adapter 3 will be used here for Host-Only networking
In the Virtual Box, VM settings for Adapter 3 -
- Enable Network Adapter - Check
- Attached to - Host-Only Adapter
- Name: VirtualBox Host-Only Ethernet Adapter #2
- Promiscuous Mode = Deny,
- Mac Address: (Write this down for later use)   eg.  ... 4A:C6
- Cable Conneceted - Check

Start the Linux VM,
- determine if it is eth0 or eth1 by: ifconfig -a
  WARNING: By default, configuration exist for eth0 only. But VB requires eth1,2,3,4
  So Copy the config file:
       cd /etc/sysconfig/network-scripts
       cp ifcfg-eth1 ifcfg-eth3
- edit /etc/sysconfig/network-scripts/eth13
    DEVICE = eth3   (change this as necessary)
    HWADDR=  enter the MAC address from the VM settings section above
    TYPE=Ethernet
    ....
    ONBOOT=yes
    NM_CONTROLLED=no
    BOOTPROTO=dhcp
- re-activate the interface:
   ifdown eth1
   ifup eth1
   /etc/init.d/network restart
- check by looking at: ifconfig -a
  the results for eth1 are: IP=192.168.39.201, Bcast 192.168.39.255, Mask 255.255.255.0
- add the lines to /etc/hosts
  192.168.39.201  VM1-host  VM1-host.domain
- add the lines to /etc/sysconfig/network
  NETWORKING=yes
  HOSTNAME=VM1-host.domain
  GATEWAY=192.168.39.1

Notes Linux

Contents

- 32bit vs 64bit
- Bitmap Fonts
- CDwriting
- WinXP and Lilo booting sequence
- SSH Tunnelling Enabling and VNC
- SSH X11 port forwarding
- USB Mounting and /etc/fstab editing
- Loading Profiles
- Linux Firewall
- Linux Runlevels
- Block web-ads
- SELinux Configuration
- For using SELinux with cgi-bin and files in Symbolic links
- RPM
- Installing Source RPMs
- Linux Commands

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

32bit vs 64bit
==============

CPU 32 bit -> integer pointers are 32bits
2 to power 32 = 4e+09   about 4 Giga stuff

Real*8 is 8 bytes - i.e. 64 bits but these are a separate thing
altogether from integer pointers.

Pointers (a C/C++ concept - alien to Fortran)
Consider an array of REAL*8 Arr(17:27)
   there are 11 elements, i.e. from Arr(17), ...., Arr(27)
   each element can be referenced with a pointer so we need 11 pointers.
   but if the pointer size is 2^32 (4e+09),
      so the max array size(w.r.t. #elements) possible is 4e09





Bitmap Fonts
============
Eg. Installing Korean Fonts

0. Log in as root

1. Download Korean Fonts and put into /scratch/chee/fonts/Korean-X
The files are:
hanglg16.pcf.gz  hanglm16.pcf.gz  hanglm24.pcf.gz

If the files are in bdf format then,
i) convert to pcf:
   bdftopcf <file>.bdf
ii) compress it
   gzip <file>.pcf
or
   compress <file>.pcf

2. Create an index file for the fonts in that directory
   mkfontdir /scratch/chee/fonts/Korean-X

this will create fonts.dir in the Korean-X directory

3. Tell the Xserver where to find the new fonts.

   xset fp+ /scratch/chee/fonts/Korean-X
   or
   xset +fp /scratch/chee/fonts/Korean-X

to check the current X settings
   xset q

to add this permanently, add the xset fp to .xinitrc

4. Refresh the Xserver setting when new fonts added if necessary

xset fp rehash

5. To view fonts in IRIX

xfd

CDwriting
=========
using cdrtools (cdrecord, mkisofs, etc..)
Ref: http://wt.xpilot.org/publications/linux/howtos/cd-writing/html/

1. Collect files and make a iso filesystem
mkisofs  -r   -o cd_image   private_collection/

where
-r   = Rockridge extns (deeper than 8 levels, permissions, etc)
-J   = Joliet extensions - more MS Windows friendly
-o   = output filename
-v   = verbose
-x   = exclude directory path
for second session only:
-C x,y = x is last session start, y is next session start
       = use "cdrecord msinfo" to get -C x,y info
       = must use with -M option
-M path = path where multi session is to merge. Usually path is
        = place normally mount CDROM eg /mnt/cdrom
EG:
cd /scratch/chee/software
mkisofs -v  -r -J -o firstImage.raw  forWinPC/AdbeRdr601_enu_full.exe

*****
mkisofs 1.12.1 (i686-pc-linux-gnu)
 61.14% done, estimate finish Fri Sep 10 16:32:27 2004
Total extents actually written = 8189
Total translation table size: 0
Total rockridge attributes bytes: 264
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used 5544
8189 extents written (15 Mb)
*****


2. Test Image
Mount the iso image (cd_image) into the directory /cdrom, so you can check the directories.

mount -t iso9660 -o ro,loop=/dev/loop0 cd_image /cdrom

3. Writing to CD
to test the CD writer and get info

cdrecord -scanbus

to write to the CD
shell> SCSI_BUS=0   # taken from listing 1 "scsibus0:"
shell> SCSI_ID=6    # taken from listing 1 "TOSHIBA XM-3401"
shell> SCSI_LUN=0
shell> cdrecord -v speed=2 dev=$SCSI_BUS,$SCSI_ID,$SCSI_LUN \
                   -data  cd_image

# same as above, but shorter:
shell> cdrecord -v speed=2 dev=0,6,0 -multi -data  cd_image

where
-dummy = simulation (laser off)
-eject = eject after burning
-v    = verbose
-multi = multi-session
-msinfo = get multi-session info
-toc = get table of contents
-data = data file (mode1 YellowBook)
-audio = audio files
speed=x  = x is speed for writing
dev = 0,0,0 for parallel.hpc.unsw.edu.au

EG
cdrecord -dummy -v speed=2 dev=0,0,0 -multi -data  firstImage.raw


4. For CDRW
writing the 2nd, 3rd sessions onwards

shell> NEXT_TRACK=`cdrecord -msinfo dev=0,6,0`
shell> echo $NEXT_TRACK
shell> mkisofs -R -o cd_image2 -C $NEXT_TRACK -M /dev/scd5 private_collection/ \

EG.
cdrecord -msinfo dev=0,0,0
   0,19591

mkisofs -v  -r -J -o secondImage -C 0,19591 -M /dev/cdrom forWinPC/Firewall

cdrecord -dummy -v speed=2 dev=0,0,0 -multi -data  secondImage


WinXP and Lilo booting sequence
===============================
   - Example for Slackware Linux

   - use the linux diskette to startup linux

   - login as root

   - modify the /etc/lilo.conf file as follow:

     lba32
     boot = /dev/hda1
     vga = 773
     image = /boot/vmlinuz
     root = /dev/hda1
     label = Slackware                <-- no space
     read-only

   - then do:

     $ lilo -b /dev/hda1              <-- lilo on hda1
     $ dd if=/dev/hda1 of=/dos/boot.lnx bs=512 count=1
     $ shutdown -r now

   - take out the linux boot diskette and re-boot the
     system to WinXP, then do:

     ** copy the file from d:\dos\boot.lnx to c:\
     ** then, modify the file c:\boot.ini to include
        following lines:

        [boot loader]
        timeout=30

        [operating systems]
        c:\boot.lnx = "Slareware Linux 9.1"
        ...

   - re-start the Wxp and there should be an option on the Wxp
     loader to boot to Linux partition.

   - If you only want to boot to the Linux partition only, just set
     the active partition to the Linux installation partition (in this
     case, it is hda1) and reboot the Windows again.


SSH Tunnelling and VNC
======================

   ssh -L x:localhost:y anotherhost

means connect to anotherhost using ssh and listen to port x on the local machine and forward any connections there to port y on anotherhost.

To start a VNC server type:

   vncserver

To start a VNC viewer type:

   vncviewer <name.of.remote.machine>:<display.number>

Using secure VNC with ssh, if a VNC server is running as display:1 on a system called linuxserver, you can start a secure SSH session by typing:

   ssh -L 5902:localhost:5901 linuxserver

So now, connecting to port 5902 on a local machine actually connects to port 5901 on linuxserver. Thus to connect from a client machine:

   vncviewer localhost:2


SSH X11 port forwarding
========================

On SSH server (remote host):
To check SSH is setup for X11 port forwarding, look at the file:
/etc/ssh/sshd_config
and see if the statement is present:
X11 Forwarding yes
then restart X11
/etc/init.d/sshd restart

On Cygwin (client):
- copy /etc/defaults/etc/ssh_config into ~/.ssh
- ensure the following are set:
ForwardX11 yes
ForwardX11Trusted yes
- DO NOT export DISPLAY=localhost:0.0

To ssh with X11 forwarding (may be different for different Unix/Linux)
ssh -X <name>@<host>



USB Mounting
=============
1. must be root to do this.
 mount -t vfat /dev/sda1 /mnt/usb/

2. Alternatively, modify /etc/fstab file to have this line
/dev/sda1        /usbdisk         vfat        user,exec            1   1

Then any user can type
  mount /usbdisk

and the files on the usb will have User's permissions

3. To unmount, type:
   umount /mnt/usb/
       or
   umounut /usbdisk


Loading Profiles
================
/etc/profile  ->  /home/<user>/.profile  ->  /home/<user>/.bashrc

Master startup scripts are in /etc :
bashrc profile csh.cshrc csh.login

/etc/profile.d   contains application specific Macro variables, eg.
java.sh java.csh



Linux Firewall
===============
/sbin/iptables --- executable
/etc/init.d/iptables                            --- script file
/etc/sysconfig/iptables                         --- configuration file

To stop firewall
/etc/init.d/iptables stop

To start firewall
/etc/init.d/iptables start

To check firewall
/sbin/iptables -L --line-numbers                --- to see various rules
/sbin/iptables -nL <INPUT/OUTPUT/RH-Firewall>   --- to see various rules and ports

Chains
/sbin/iptables -F [chain]                       --- deleting all rules in the chain
/sbin/iptables -X [chain]                       --- delete optional user-def chain
/sbin/iptables -Z [chain]                       --- to reset / zero packet and byte counter
/sbin/iptables -N [chain]                       --- to create new user-def chain

<Target>: {Accept, Queue, Drop, Return}

Rule-specifications
/sbin/iptables -D chain rule                    --- delete rule in chain
/sbin/iptables -A chain rule                    --- append rule to end of chain
/sbin/iptables -I chain [rulenum] rule          --- insert rule at rulenum or 1st position of chain
/sbin/iptables -R chain rulenum rule            --- replace rule to at rulenum positin of chain

Rules specs: -<s/d> [!] IPaddr[/mask] [!] [port[:port]]
   !=negate, []=optional

Other Parameters
-i           --- interface, eg eth0
-p           --- protocol, eg tcp, udp, icmp, all
-s           --- source (see above)
-d           --- destination (see above)
-j           --- jump <target>
-m state     --- match with the state module
--line-numbers

<James Sainsbury help>
Example - Webserver httpd - open at port 80
iptables -A INPUT -p tcp -i eth0 --dport 80 -j ACCEPT - Open HTTP port

  or

So locate the rule in RH-Firewall-1-INPUT for WWW
eg    iptables --line -L RH-Firewall-1-INPUT

eg    9  ACCEPT  tcp  --  anywhere  anywhere  state NEW tcp dpt:http
and insert a rule just after that rule (9 here) so insert before rule 10

# iptables -I RH-Firewall-1-INPUT 10 -p tcp -s 0/0 -d enrol1 -dport 8080 -m state --state NEW -j ACCEPT
If you are satisfied the rule is working save the ruleset with
    /etc/init.d/iptables save
If you trash the ruleset but haven't saved just
    /etc/init.d/iptables restart
(You will notice there is a preceding rule
    7  ACCEPT  all  --  anywhere  anywhere  state  RELATED,ESTABLISHED
which accepts incoming packets as part of an ongoing connection.)
</James Sainsbury help>


To Save iptables:
/sbin/service iptables save



My Example for opening ports on local enrol1 to communicate with BIS on eb
iptables -I RH-Firewall-1-INPUT 12 -p tcp -s eb -d enrol1 --sport 49153:49155 -m state --state NEW -j ACCEPT


- To redirect all external traffic coming into port 80 into port 8080:
1. Use firewall configuration software to accept all TCP traffic from destination port 8080
2. (as root): iptables -t nat -A PREROUTING -p tcp --dport 80 -i eth0 -j REDIRECT --to-port 8080

- To redirect internal traffic coming into port 80 to port 8080
1. use program called rinetd.
2. in the /etc/rinetd.conf, have the line
"xx.xx.xx.xx 80 xx.xx.xx.xx 8080"
3. start the rinetd service



Linux Runlevels - Auto Startup
==============================
/usr/sbin/xxxd              --- daemon or startup file to run
/etc/xxx.conf               --- configuration file of the progran
/var/run/xxxd.pid           --- process ID of the file that run

Runlevels
0   - system shutdown
1   - single user, no networking or daemons, for maintenance
2   - multiuser but no NFS, can access network but others cannot get in
3   - normal multiuser, no X login
4   - undefined
5   - multiuser X Display manager, X login from consoles
6   - reboot
s/S - single user mode

/etc/inittab            - shows how the system is initialized
/etc/rc.d/init.d/         - list of daemon files, used or not used in startup
/etc/rc.d/rc3.d/        - links to actual files in init.d, at this particular runlevel 3


Program Sequence       Config file
at Startup
===========================================
/sbin/init             /etc/inittab

/etc/rc.d/init.d/*     /etc/sysconfig/daemons

/etc/rc.d/rcN.d/*
(Sym links of init.d)


Note the files in rcN.d are preceded by K or S and a number
K = kill job
S = start job
number = order of pograms

To check what services are running:
/sbin/chkconfig --list
    service <service name> status

- Remember to check Linux Firewalls if network services do not start

- edit script files in /etc/init.d/ to contain
1. start/stop functions (and others functions restart)
2. commands in start function MUST NOT BE EXECUTABLE commands. They
   should be executable scripts.
3. use daemon function to run as non-root user
4. DO NOT USE & (forking) at the end of command
5. Example:
        daemon --user=<userName> <ShellScriptName> 1> job.log 2>&1

- To register
/sbin/chkconfig --level 5 initdScript on










Block web-ads
==============

Most of us can't stand getting pummeled by ads everytime we go to a website. There
are a couple of ways to stop these ads.  One is to edit your /etc/hosts file and list
 your loopback address, 127.0.0.1, with the ad hosting address. For example, to
block ads from ar.atwola.com open up /etc/hosts as root:

[root@blueox]# vim /etc/hosts

Next, add the following line:

127.0.0.1 ar.atwola.com

This will line will tell your browser to look on 127.0.0.1 for the ads from
ar.atwola.com.  Since localhost doesn't have the ad, it won't be displated.
You can download a hosts file that has a multitude of ads to block.
Personally I DO NOT RECOMMEND THIS approach.  I have noticed that
my computer slows down trying to search through the hosts file.

The RECOMMENDED approach for firefox and mozilla users is to go to
the chrome directory of your personal browser folder.

[lumbrjackedpcj@blueox]# cd ~/.mozilla/firefox/dxshwykf.default/chrome

The directory dxshwykf.default will be different for you installation.

1. First, create the file userContent.css

[lumbrjackedpcj@blueox]# touch userContent.css

2. Next, edit this file with your favorite text editor

[lumbrjackedpcj@blueox]# vim userContent.css

3. Now, following the instrunctions on http://www.mozilla.org/support/firefox/adblock,
  copy this userContent.css and paste it in yours.



SELinux Configuration
======================
Intro
The latest releases of Linux distributions incorporate the mandatory access
control architecture SELinux of the NSA into the Linux kernels.
More information on SELinux can be found at the following link:
http://www.nsa.gov/selinux/info/faq.cfm

/usr/sbin/sestatus -v                      --- to get status of selinux

En/Disabling SELinux:
1 - in your /etc/sysconfig/selinux file, set:
SELINUX=<disabled|enforcing|permissive>
... this is not TRUE disable, it only switches off enforcing and skips loading policy.

2 - if using GRUB as bootloader
go to /boot/grub/grub.conf or menu.lst (symlink to grub.conf)
add this line to the correct boot section
selinux=<0|1>

3 - if using LILO (... to be completed ..., contributions welcome)


Error Message:
"error while loading shared libraries: /usr/lib/liba52.so.0: cannot restore segment prot after reloc: Permission denied"
then your SELinux configuration is preventing your application from launching.


Solutions:
For shared libraries failing to load, try:
chcon -t texrel_shlib_t /usr/lib/liba52.so.0
For some other libraries
chcon -c -v -R -u system_u -r object_r -t lib_t $InstallationDirectory/SystemFiles/Libraries/


# View file/directory contexts: ls -Z /var/www
# Set file/directory context:
    chcon -R -h -t httpd_sys_content_t /home/user1/public_html
Options:
    * -R: Recursive. Files and directories in current directory and all subdirectories.
    * -h: Affect symbolic links.
    * -t: Specify type of security context.
# View security context of running processes: ps -eZ
# View security context of a user: id
    uid=500(user1) gid=500(user1) groups=500(user1) context=user_u:system_r:unconfined_t
# Security contexts for a new file default to that of the parent directory.


For using SELinux with cgi-bin and files in Symbolic links
============================================================
Example:
perl script:      /var/www/cgi-bin/overview
Data file:        /home/BIS/BIS/log/ANGLE.txt
Symbolic Link:    /var/www/html/BIS/ANGLE.txt

The security context, especially the subdirectories that contain it are
as follows:

/var/www/cgi-bin/overview      root:object_r:httpd_sys_script_exec_t

/var/www/html/BIS              user_u:object_r:httpd_sys_content_t
/var/www/html/BIS/ANGLE.txt    user_u:object_r:httpd_sys_content_t

/home                          system_u:object_r:home_root_t
/home/BIS                      user_u:object_r:user_home_dir_t
/home/BIS/BIS                  user_u:object_r:httpd_sys_content_t
/home/BIS/BIS/log              user_u:object_r:httpd_sys_content_t
/home/BIS/BIS/log/ANGLE.txt    user_u:object_r:httpd_sys_content_t

Note that it is enough to ensure the "type" context at the /home/BIS/BIS
level and its contents seem to inherit the type.


RPM
===
Redhat package manager

Query
rpm -q -a                        --- list all packages
rpm -q -p <package_path.rpm>     --- list packages inside package_path rpm file
rpm -q -l -p <package_path.rpm>  --- list all files inside package_path rpm file
rpm -q -a -i                     --- list info on all packages
rpm -q -a -f                     --- list files of all packages



Installing Source RPMs
=======================
1. Get source code in rpm form from the "SRPMS" or "SOURCE" directories
   in common repositories
http://download.fedora.redhat.com/pub/fedora/linux/core/5/source/SRPMS/

2. As root, unpack the source RPMs
   rpm -ivh example.src.rpm

.... this will put the patches in /usr/src/redhat/SOURCES. But these
are not the real working file.
.... there is also a build *.spec file put in /usr/src/redhat/SPECS
 
3. Build the source code of the packages using:
 command: rpmbuild -bp --target=<arch> kernel.spec
      # cd /usr/src/redhat/SPECS/
      # rpmbuild -bp --target=i686 kernel-2.6.spec
.... the source may appear in /usr/src/redhat/BUILD

4. Modify and recompile source
5. Package into rpm using:

Linux Commands
===============

XConfuration:
/usr/sbin/mouseconfig  ------- mouse
chsh ------------------------- to change default shell
grep chee /etc/passwd -------- tells you what shell you are from

Xconfigurator ---------------- to use XFree86
/etc/X11/XF86Config ---------- configuration file
ls /usr/lib/X11/rgb.txt ------ to see a list of X11 defined color names


/usr/sbin/timeconfig --------- change time

/usr/sbin/ntsysv ------------- choose program to
/sbin/chkconfig -------------- boot during startup

/usr/doc/HOWTO --------------- HowTo help files


Commands:
useradd **** ----------------- add user with name ****
passwd ***** ----------------- make password for user
pwd -------------------------- print working directory
su **** ---------------------- to change to another user (from superuser)
su --------------------------- superuser
/sbin/shutdown -r now -------- shutdown and reboot now
startx ----------------------- to start X windows
ps -e/f/a/l/u ---------------- process
mount /dev/fd0 /tmp ---------- mount 1st floppy drive to /tmp
umount /tmp ------------------ umount any device from /tmp
cat notes >> notes2 ---------- prints the contents of a file to standard output
      and then redirects and APPEND to file "notes2"
head [-count] *****       list first few lines of file
tail [-/+ [startline] [bcl]] ***  list the last few lines
wc **** ---------------------- count the number of words, lines etc in a file
alias l='ls -al' ------------- make an alias, eg. to list files
ln **file1** **file2** ------- link file2 to original file 1 (hard link)
ln -s **file1** **file2**----- do symbolic link
ls -i ------------------------ list file and look and their i-node to see linked files
rsync -auv src dest ---------- copy / update between local/remote src/dest
size ------------------------- to calculate the size of the program
wc -l/c/w **file** ----------- Word Count lines, characters, words in a file
tr --------------------------- Translate, delete or replace text in a file.
tar cf - . | zip name -  ***** Tar and then zip
unzip -p backup | tar xf - *** reverse of above - ie. unzip and untar
script <filename> ------------ starts a shell - records the whole sesion - exit the shell to stop script


Help Commands:
man -k **** ------------------ to search for keywords in manual entries
man # intro ------------------ #={1..8}, 8 intro topics of Unix manual.
man -M /opt/lam/man mpirun --- manual for "mpirun" program installed in /opt/lam
rpm -qdf /usr/bin/rtin ------- query
zless /usr/doc/HOWTO/...gz --- read compressed howto files without uncompressing
locate -d <dbfile> <filename>- find a command or file called filename
updatedb --localpath=<dirPath> --output=<dbfile>
                               to update data file before using locate
                               dirPath=path of file system to put in db
                               dbfile=database file to log info to.
slocate ......                 replaces updatedb and locate commands
/etc/cron.daily/update ------- create database before using locate
find / -name .... ------------ find the file named ....
find ~ -size 0 -print -------- find file with zero size
crontab -e ------------------- to edit the crontab for current user

Build / Compile Programs
./configure --prefix=/home/che sets the installation to your local dir so no need root permissions

Environment:
set -------------------------- to list enviroment variables
xport AUTO=VW ---------------- set the variable auto=vw
alias ls='ls -color=auto' ---- colorize ls .. put this in .profile or .bashrc

./a.out ---------------------- to execute the file "a.out"
. <command/script> ----------- execute command or script in CURRENT environment

Process Control:
kill -9 PID ----------------- sure kill the process
nohup **** ------------------ No hang-up for process ****
nice -# **** ---------------- reduce priority of command
nice --# **** --------------- increase priority of command
sleep **** ------------------ timer - stalls for certain time
at -l/r/m time date **** ---- timer - run command AT specified TIME

Compilers
cc -------------------------- SGI MIPS c compiler
CC -------------------------- SGI MIPS c++ compiler
gcc ------------------------- gnu c compiler
g++ ------------------------- gnu c++ compiler
insight --------------------- online help manual on SGI MIPS C/C++ compilers

Gnuplot
load "plotfile" ------------- runs the script file "plotfile"
set term postscript color
set output "file.ps" -------

Emacs
<shift> <left-mouse> -------- change font size
.Xdefaults ------------------ change emacs properties
M-x replace-strings --------- to replace all occurances of strings
C-q C-m --------------------- produces the letters "^M"
M-x replace-strings, C-q C-m  dos2unix file conversion within emacs
C-c C-w --------------------- Fortran End of Column marker
C-c C-r --------------------- Fortran ruler
M-C-j ----------------------- Fortran continuation line


Browsers:
netscape -install ----------- make colour look better

Graphics
gif2ppm --------------------- convert gif to portable pixmap format
pnmtogif -------------------- convert any pixmap to gif
pstoimg -transparent -------- make background transparent

Secure Operations
scp newfile chee@parallel.hpc.unsw.edu.au:~/programs

Networking
nslookup www.google.com ----- returns the IP address of the URL
ping ------------------------
ifconfig <ethernet card> ---- configuration of adapter
netstat -in -----------------
echo 1 >/proc/sys/net/ipv4/ip_forward -- allow routing
echo 1 >/proc/sys/net/ipv4/conf/all/rp_filter -- prevent IP spoofing

System Admin / Setup
/etc/passwd ----------------- user details
/etc/sys_id ----------------- name of the system
/etc/hosts ------------------ list of IP address and Names
~/.rhosts ------------------- list of computers and usernames for rsh without password
background ------------------ customizing background of screen
ls -l /lib/libc-*  ---------- check the glibc version
uname -r   ------------------ check the kernel version
/etc/rc.d/rc.local ---------- startup programs are included here

System /Hardware Information
lsmod ----------------------- list loaded modules
lsdev ----------------------- display info about installed hardware
modinfo --------------------- information about installed modules
free ------------------------ display free/used memory on the system
more /proc/interrupts ------- view IRQ channels
more /proc/dma -------------- view dma channels
more /proc/ioports ---------- view I/O memory addresses

Change Resources
ulimit -a ------------------- check user limit
ulimit -aH ------------------ check user HARD-limit
ulimit -s 32678 ------------- set SOFT & HARD stack limit
ulimit -sS 32678 ------------ set SOFT stack limit
echo 8192 >/proc/sys/fs/file-max   --- increase number of files openable
echo 16384 >/proc/sys/fs/inode-max --- increase number of files openable

File Commands
file ***** ------------------- prints the type of the file ****
find -c ---------------------- find file which has changed
stat -t ---------------------- provides info about file
diff file1 file2 ------------- output differences between files

Remote X Gui Interface
startX ----------------------- starts the X server
startkde --------------------- starts the KDE desktop

Modules
insmod ----------------------- insert module quick UNSAFE way
modprobe --------------------- insert module SAFE way

Notes RStudio Server

Notes RStudio Server
=====================

TO DO
Details
Connecting to Linux
Basic Linux Admin
R installations on Linux
R Studio Server Installation
Installing Apache
Configuring Apache Daemon - httpd
R Studio Management / Configuration
Configuring Subversion for Rstudio
Running R-Studio with Subversion
Services That Need To Be Restarted



Details
========
server names, login names, etc --- see details.txt


Connecting to Linux
=====================
Get Open Source SSH software
http://www.bitvise.com/tunnelier-download

To connect via SSH:
1. Open Bitvise Tunnelier
2. Host: <LinuxServer>
   Port: 22
   Username: <LinuxUser>
   Initial Method: password
   Password: <>
3. Click on Open New Terminal Console.
4. To check sudo password is working:
   sudo more /tmp/dsm.sys.bak
   ... or use any files that can be viewed by root only.


Basic Linux Admin
===================
To check which version of GCC goes with which version of RH Linux.
http://distrowatch.com/table.php?distribution=redhat

check version of Linux
   cat /proc/version
Linux version 2.6.18-308.el5 (mockbuild@x86-007.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Jan 27 17:17:51 EST 2012
   uname -a

check 32bit vs 64 bit
   uname -m
----- output ------
x86_64
file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped

check packages
yum list installed
yum list available

For X windows:
sudo vi /etc/ssh/ssh_config
   - modify this file so that
   X11Forwarding yes
   X11DisplayOffset 10
   X11UseLocalhost yes

No  DISPLAY=myws:0; export DISPLAY
No  DISPLAY=127.0.0.1:0.0; export DISPLAY
No  DISPLAY=:0.0; export DISPLAY
No  DISPLAY=:0; export DISPLAY
No  DISPLAY=.0; export DISPLAY

In Remote box, type:
    DISPLAY=localhost:10.0; export DISPLAY
xterm &

To avoid setting DISPLAY everytime, edit .bash_profile by adding these lines:
    DISPLAY=localhost:10.0
export DISPLAY


Fedora, and perhaps RHEL, rpm packages are "likely" to be installed in:
/lib/
/usr/bin/
/usr/lib/
/usr/share/doc/
/usr/share/man/

top             - to see status of processes, press 'q' to quit, press 'H' to toggle threads view.
                - press 'n' to enter the number of lines displayed.
mpstat -P ALL   - to see stats on all CPUs

/etc/services   - a file with list of port numbers and their associated services.
netstat         - to see the ports of active services


R installations on Linux
=============================
 http://cran.r-project.org/doc/manuals/R-admin.html
 http://cran.r-project.org/doc/manuals/R-admin.html#Getting-and-unpacking-the-sources

Manual Downloads:
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R.html
R-2.15.0-1.el5.x86_64 [16 KiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R-core.html
R-core-2.15.0-1.el5.x86_64 [36.0 MiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/R-devel.html
R-devel-2.15.0-1.el5.x86_64 [90 KiB] Changelog by Tom Callaway (2012-03-30):- Update to 2.15.0
http://dl.fedoraproject.org/pub/epel/5/x86_64/repoview/xdg-utils.html
xdg-utils-1.0.2-4.el5.noarch [52 KiB] Changelog by Lubomir Kundrak (2008-01-25):- Fix for CVE-2008-0386 (#429513)

rpm -i R-core-2.15.0-1.el5.x86_64.rpm
warning: R-core-2.15.0-1.el5.x86_64.rpm: Header V4 DSA signature: NOKEY, key ID
217521f6
error: Failed dependencies:
        cups is needed by R-core-2.15.0-1.el5.x86_64
        libtk8.4.so()(64bit) is needed by R-core-2.15.0-1.el5.x86_64
        tetex-latex is needed by R-core-2.15.0-1.el5.x86_64
        xdg-utils is needed by R-core-2.15.0-1.el5.x86_64

Type these commands to install dependencies:
sudo yum install cups
sudo yum install tetex
sudo yum install tk.x86_64
sudo rpm -i xdg-utils-1.0.2-4.el5.noarch.rpm
sudo yum install tetex-latex
sudo rpm -i R-core-2.15.0-1.el5.x86_64.rpm

To check rpm's are installed, eg.
sudo rpm -q -i xdg-utils
sudo yum info R-core-2.15.0-1.el5.x86_64     # can yum even if packaged was RPM

To check yum's are installed, eg.
sudo yum info tetex-latex
sudo more /var/log/yum.log

yum configuration details are in:
/etc/yum.conf
/etc/yum.repos.d/*.repo





R Studio Server Installation
==============================
Ref: http://rstudio.org/download/server

1. Extra Packages for Enterprise Linux  (EPEL)
Download epel-release-5-4.noarch.rpm from:
http://dl.fedoraproject.org/pub/epel/5/x86_64/

(Transfer that file to the Linux box)

In the Linux box, do (see WARNING below first) the following:
sudo rpm -Uvh epel-release-5-4.noarch.rpm

To check installation:
sudo rpm -q -i epel-release-5-4

WARNING: It may not be necessary to do this step. Not sure what the consequences of avoiding this step is. The main problem with DOING this step, from actual experience, is that it corrupts the yum repository. Actually it added 2 repository which cannot be accessible. The solution is to move the files below to somewhere else:
sudo move /etc/yum.repos.d/epel*  <Some Junk Directory>



2. Install R (core or base) package.
Since R is installed in the step above, we just need to check R is installed by:
     sudo yum info R-core-2.15.0-1.el5.x86_64     # can yum even if packaged was RPM
OR
     sudo rpm -q -i R-core


3. Installing R Studio
Download the Rstudio package from: http://download2.rstudio.org/rstudio-server-0.95.265-x86_64.rpm

Transfer the rpm to the Linux box if not there already.

Install:  sudo rpm -Uvh rstudio-server-0.95.265-x86_64.rpm

Check:  sudo rstudio-server verify-installation
   ...the following output from the check still allows Rstudio to run through web in the end - so it is OK.
rserver[20481]: WARNING R include path (/usr/include/R) not found; LOGGED FROM: bool core::r_util::<unnamed>::validateREnvironment(const core::r_util::EnvironmentVars&, const core::FilePath&, std::str
ing*) /root/rstudio/src/cpp/core/r_util/REnvironmentPosix.cpp:379
Starting rstudio-server:                                   [  OK  ]


Configuring for External Libraries - WORK IN PROGRESS
- This are R libraries that are not part of the standard R base. They are developed by other users, can be downloaded from the R repository.
- For R studio, let only the svn user have control over the R libraries and they will be stored at:
   /home/svn/R/library
- Create the configuration files as follows, edit the file:
sudo vi /etc/rstudio/rserver.conf  
- Put this content in the file:
rsession-ld-library-path=/home/svn/R/library
- Test and restart server
 sudo rstudio-server test-config
 sudo rstudio-server restart

 Package Management Proposal
USE CASE 1: A normal R user wants to install packages using Rstudio-server GUI.
 - Problem 1: it appears each user need to install their own packages under ~/R/library
 - Problem 2: Linux server cannot access outside world.
 - Solution:
 a. Create a local repository on Linux box, so that Rserver points to http://localhost/src/contrib/
 b. On Windows: Manually or write a Web Robot to source all updated packages. Then transfer all packages to Linux server under <Web>/src/contrib
 c. On Linux, each user specify the localhost as R repository.
 Summary, essentially we run our R Repository mirror. Each user install packages as they need. All users of CBA are guaranteed to use the same version of packageas because they all source it from the same CBA repository.
 d. An extra step may be to link all users directories ~/R/library to a common directory on Linux server. This means when one user install a package, it will be available to all users.

USE CASE 2: An R administrator installs packages for everyuser.
The R administrator gets the R library SOURCE, in the form of *.tar.gz and puts in them in <LOCAL_REPO>.
To install individual packages, the R administrator will do:
    sudo R CMD INSTALL -l <R_SHARE> <LOCAL_REPO>/adk_1.0-2.tar.gz

... where the example package to install is adk, and the location to install may be <R_SHARE>="/usr/share/R/library"
Check that R_SHARE is listed as one of the directories given by R command: .libPaths()

To remove:
sudo R CMD REMOVE -l <R_SHARE> adk

To Check:
- login as a regular R user and type:
    library(help="adk")

With this method, any R user can use the package and all users will be using the same version.
The problem of accessing the repository still remains and has the same solution described as above.


Some R package management commands:
packageStatus()$inst
getOption("repos")                 - list R repositories
getOption("defaultPackages")       - list packages loaded by default
remove.packages(c("pkg1", "pkg2"), lib = file.path("path", "to", "library"))
.libPaths()    - default libpaths
     "/home/cheec/R/library"             "/usr/lib64/R/library"            
     "/usr/share/R/library"              "/usr/lib/rstudio-server/R/library"
Sys.getenv("R_LIBS_USER")                -  "~/R/library"  *** note these are R env vars - they are not Linux env vars
Sys.getenv("R_HOME")                     -  "/usr/lib64/R"
R_HOME/etc/repositories                  - list of directories
R CMD INSTALL -l <LIB> pk1 pk2           - installs packages pk1, pk2 to location <LIB>
install.packages(c("pk1", "pk2"))        - within R, install packages pk1, pk2
install.packages("pk1", dependencies=TRUE)  - within R, install packages pk1 and dependencies.
install.packages("~/R/library/adk_1.0-2.tar.gz", repos = NULL)
remove.packages("adk")
library(help="adk")                      - to check if adk has been installed correctly
update.packages()                        - checks and updates all necesary packagees
packageStatus()$inst
R CMD check -l ~/R/library ~/R/library/adk_1.0-2.tar.gz     - to check and INSTALL the package adk is installed.
sudo R CMD INSTALL -l /usr/share/R/library ~/R/library/adk_1.0-2.tar.gz
sudo R CMD REMOVE -l /usr/share/R/library adk


To set up a local CRAN mirror:
http://cran.r-project.org/src/contrib/
http://cran.r-project.org/mirror-howto.html


http://r.789695.n4.nabble.com/R-2-6-0-packages-installation-through-a-proxy-not-working-td837190.html




Installing Apache
===================
Download http://apr.apache.org/download.cgi
apr1.6
apr-util 1.4.1
apr-iconv1.2.1
Download http://mirror.overthewire.com.au/pub/apache//httpd/httpd-2.4.2.tar.gz

Transfer the *.gz files to the linux box.

Extract the Apache source:
    sudo mv <the FOUR *.gz files>  /usr/local
cd /usr/local
    sudo tar -xzvf <the FOUR *.gz files>

Installing apr:
   go to the appropriate directory, eg cd /usr/local/apr
   sudo ./configure
   sudo make
   sudo make install

Installing apr-util
   go to the appropriate directory, eg cd /usr/local/apr-util....
   sudo ./configure --with-apr=/usr/local/apr
   sudo make
   sudo make install

Install pcre-devel
   sudo yum install pcre-devel.x86_64

Installing apache2
   go to the appropriate directory, eg cd /usr/local/httpd...
   sudo ./configure --with-apr=/usr/local/apr --enable-proxy --enable-proxy-http --enable-proxy-html  --enable-xml2enc
   sudo make
   sudo make install


Uninstall apache2 - only if you need to uninstall for whatever reasons
   go to the appropriate directory, eg cd /usr/local/httpd...
   sudo make clean
   sudo make distclean
 
Installing elinks - a text based browser (for checking whether webserver running locally)
   sudo yum install elinks

Configuring Apache - to use Reverse Proxy to Rstudio-Server
- apache src files are at: /usr/local/httpd....  --> <SRC_DIR>
- by default, apache is installed under : /usr/local/apache2    (check this by <SRC_DIR>/configure --help)
- from here on, let notation  $PREFIX="/usr/local/apache2"
- to start   apache server:  $PREFIX/bin/apachectl -k start
- to stop    apache server:  $PREFIX/bin/apachectl -k stop
- to restart apache server:  $PREFIX/bin/apachectl restart
- to check if apache server is running: pstree   (look for httpd in text)
- to check apache installation, edit the file $PREFIX/htdocs/index.html and write some html content like "Hello World"
  open up elinks text browser on the linux box, and goto http://localhost, you should see the index.html modified above.  
  open up normal browser on your PC desktop within CBA network, and goto http://<HOST>, you should see the index.html modified above.  


Configuring Proxies
  References
- http://www.apachetutor.org/admin/reverseproxies
- http://thelowedown.wordpress.com/2008/10/12/reverse-proxy-with-apache/
- http://code.google.com/p/puppeteer/wiki/ProxyConfiguration
- http://wiki.apache.org/httpd/ProxyAbuse

Put the following code into PREFIX/conf/httpd.conf file:
<VirtualHost *:80>
<Proxy *>
Allow from localhost
</Proxy>
ProxyPass        / http://localhost:8787/
    ProxyPassReverse / http://localhost:8787/
</VirtualHost>

- stop apache server (see above)
- start apache server (see above)
- Open a browser on your local PC desktop and goto http://<HOST> (no port numbers needed). You will be REDIRECTED to R-Studio.


Configuring Apache Daemon - httpd
===================================
Note there are TWO versions of apache. One is the RPM package that came prebuilt with RHEL. The other one is the one manually built using the method above. The self-build one is needed over the RPM one because several specific options for Rstudio requires this.

The RPM apache is configured to be ready to start up with the following configuration.
File: /etc/rc.d/init.d/httpd                 - startup script that has the following relevant information
- points to /etc/sysconfig/httpd             - has httpd system related information
- points to /usr/sbin/apachectl              - control program
- points to /usr/sbin/httpd                  - actual daemon program
- points to /var/run/httpd.pid
- points to /var/lock/subsys/httpd
- points to /etc/httpd/conf/httpd.conf       - main server configuration file

For the apache build locally, it is not set to startup, but has the following files
- /usr/local/apache2/bin/apachectl           - control program
- /usr/local/apache2/bin/httpd               - actual daemon program
- /usr/local/apache2/conf/httpd.conf         - main server configuration file

*** You will be SUPPLIED / GIVEN a file called httpd_start
The idea is to modify the already existing /etc/rc.d/init.d/httpd  (from RHEL's Apache rpm) so that it has the contents of the given httpd_start file
1. Edit the file /etc/rc.d/init.d/httpd  so that its contents are identical to httpd_start.
OR
2.Alternatively to copy-paste is to upload httpd_start to directory  /etc/rc.d/init.d/ to REPLACE existing httpd file.
3.Also do this to hide another file:  sudo mv /usr/sbin/httpd  /usr/sbin/httpd_REDHAT_old


to check that this new httpd is working, do the following:
cd /etc/rc.d/init.d
sudo ./httpd start
--- check that the webserver is running ----
sudo ./httpd stop


Then do the following:
sudo chmod 755 /etc/rc.d/init.d/httpd        - to ensure it has permissions to start
sudo chkconfig --list                             - do this to check that httpd is registered or not
sudo chkconfig --add /etc/rc.d/init.d/httpd       - to register for startup, if not there yet
sudo chkconfig --level 2345 httpd on              - to switch on at specific runlevels



Ref - for init.d/httpd startup script
http://www.techiecorner.com/104/how-to-auto-start-apache-during-boot-time-linux/
http://www.zrinity.com/developers/apache/apache2src.cfm


R Studio Management / Configuration
====================================
   To manually stop, start, and restart the server you use the following commands:
sudo rstudio-server stop
sudo rstudio-server start
sudo rstudio-server restart

   To list all currently active sessions:
sudo rstudio-server active-sessions
   To suspend an individual session:
sudo rstudio-server suspend-session <pid>
   To suspend all running sessions:
sudo rstudio-server suspend-all
   The suspend commands also have a "force" variation which will send an interrupt to to the session to request the termination of any running R command:
sudo rstudio-server force-suspend-session <pid>
sudo rstudio-server force-suspend-all
   The force-suspend-all command should be issued immediately prior to any reboot so as to preserve the data and state of active R sessions accross the restart.

   Taking the Server Offline - If you need to perform system maintenance and want users to receive a friendly message indicating the server is offline you can issue the following command:
sudo rstudio-server offline
sudo rstudio-server online
   These two commands are independent of start stop. When suspended, even if the server is restarted, rstudio is still not accessible (ie offline), until it is switched back to online again.

Workflow for taking R offline:
Method A
- on server: sudo rstudio-server offline
- on user PC: 1. "Error: Status code 503 returned" display in the R session 2. "Rstudio Temporarily Offline" dialog appears. 3. User cannot do anything.
- on server, this has to be done IN ORDER, otherwise session will not start.
   1. sudo rstudio-server restart
   2. sudo rstudio-server online




Version Control using Subversion
===================================
RHEL5 has got subversin 1.6.11 pre-installed for both i386 and x86_64
To use Subversion with R, the work needs to be organized a Rstudio Projects.

1. To check if the svnserver starts automatically, whether svn is preinstalled for Red Hat, or self install later, type:
  ls -laF /etc/init.d/svnserve
to see if the file exists. This file runs a script to start the daemon.

2. Create svn user and group, type:
sudo useradd svn  - use id svn to check user details
sudo passwd svn   - make password for svn -> "svn"
sudo usermod  -a -G R_POC_TEAM svn    - this puts the user "svn" to the group called R_POC_TEAM.

Ensure that all other user of svn is in the same group called "R_POC_TEAM". To check which groups the users are in, type:
sudo more /etc/passwd

3. Create the SVN repository for all members of Group Quantitative Analytics (GQA).
- this is done only ONCE, when a new VM and subversion is installed.
- all users and the 'svn' user need to be in the same group. In this example, all these users belong to a group called R_POC_TEAM. If this is not true, make it so.
- Login as svn user.
- create the repository called QuantAnalytics, type:
svnadmin create --fs-type fsfs /home/svn/QuantAnalytics

This creates the repository QuantAnalytics, that will contain all projects. Note that --fs-type fsfs is the preferred filesystem for svn rather than the Berkley DB filesystem (see manual).

4. Check the file /etc/services. If it does not contain the following lines, then add them:
svn    3690/tcp     # Subversion
svn    3690/udp     # Subversion


5. Configuring Authentication and Authorization
- LOGIN AS USER called 'svn'
- go to the QuantAnalytics repository, eg:
    cd /home/svn/QuantAnalytics
- Create a file under ...QuantAnalytics/conf/svnserve.conf, with the following content:
[general]
password=db = passwd
realm = QA realm
anon-access = read
auth-access = write
...... passwd is actually the word "passwd".

- Ensure the file called 'passwd' exist under ...QuantAnalytics/conf/ and that it has the following content:
[users]
harry = harryssecret
sally = sallyssecret

.... where harry and sally are valid users in the linux box.


6. Hide the absolute path of the svn server and provide relative paths to user only.
Edit this file: /etc/rc.d/init.d/svnserve

by adding "-r /home/svn" to the following line if it exist, so that it becomes:
args="--daemon --pid-file=${pidfile} $OPTIONS  -r /home/svn"

Now when the svn server is mentioned, it is done by:
<svn server>/QuantAnalytics  instead of  <svn server>/home/svn/QuantAnalytics






The following information are for references:
- Access SVN. The URL pointing to the repository is:
svn://<HOST>/QuantAnalytics
svn://localhost/QuantAnalytics

- Some svn admin commands: svnadmin, svnlook, svndump, svndumpfilter, svnsync

- svnlook info <repos>      - prints information for the repository at <repos>.
                            The information include author, date, number of lines of log, log message

- sudo killall svnserve     - force termination of any active svnserve services. Also solves the error with message:
                            " svnserve: Can't bind server socket: Address already in use"

- sudo /etc/init.d/svnserve start       - to start svn server once off.

***** How to setup iptables - NOT Needed here
http://articles.slicehost.com/2007/9/5/introduction-to-svnserve




Running R-Studio with Subversion
==================================
WARNINGS
A. NEVER NEVER NEVER have filenames with SPACES or ANY OTHER symbols except . (fullstop) and _ (underscore).

1. To run Rstudio, open a browser and type this URL:
         http://<HOST>

To login as another user (for administrators only), when the user name field is no longer visible, type:
http://<HOST>/auth-sign-in

The sign in password is the same password of the Unix account.

2. Go to Tools - Options, click the Version Control icon on the left navigation pane.
- Under "SVN executable:", type:  /usr/bin/svn

3. The Repository URL is:
svn://localhost/QuantAnalytics

4. To Finish R, there are two ways:
- Within Rstudio, click File - Quit R.
This will finish off the current R but immediately asks you if you want to start a new R version.
- On the top right, click Sign-Out
This will sign you out, but your session is saved on the server. When you login again, you will see the same session where you left off, including all the variables in the workspace will still be there.
- Also while working in a project, MUST close project before exiting R, otherwise the same project opens automatically when Rstudio is started. To close project, click: Projects - Close Projects.


5. (Recommended) To create a new project and at the same time, enabling version control.
This is the recommended way, even if you have existing R code files and existing project, to simplify the process of putting your code into version control, just create a new project associated with version control. Then copy your files into this new project.
- Once logged in to Rstudio, click Tools - Shell
- Type: mkdir ~/<ProjName>
  where ~ means your home directory, <ProjName> is the name of your project
- Type:  svn import ~/<ProjName> svn://localhost/QuantAnalytics/<ProjName>
- Click Close, to get back to main Rstudio
- Click Project - New Project
- Choose "Version Control" from the New Project dialog.
- Click "Subversion", not "Git"
- Enter the following details:
Repository URL:                        svn://localhost/QuantAnalytics/<ProjName>
Username:                              <your Linux account username>
Project Directory Name:                <ProjName>
Create project as subdirectory of:     ~
- Click "Create Project"
- RStudio will open up in the new project directory. Experiment a bit here. Create a file, write something, save it.
- Copy files from other directories into this directory. Then add these files to version control.


6.A. When a Directory for a group of R files exist, to put it in version control, first formalize that directory as a R project, then add to version control.
- Once logged in to Rstudio, click Project - New Project
- Choose "Existing Directory" from the New Project dialog.
- In the Project working Directory, enter the existing directory path, eg. ~/ExistingDir
- Click "Create Project".
At the end of this stage, a new project has been created to encompass old files in an old directory.

6.B. To put existing R project directory into SVN:
- Login to Rstudio and open the project created in the previous step.
- Check that Rstudio is now in that project directory, type:    getwd()  
- On the Menu, click Tools - Shells
- To upload files to SVN, type:
(In General) svn import <local path> <SVN repo URL> -m "your message"
(Example)    svn import ~/myProj svn://localhost/QuantAnalytics/myProj -m "Initial Import"

svn delete  svn://localhost/QuantAnalytics/myProj/.Rproj.user
... then press C
svn delete  svn://localhost/QuantAnalytics/myProj/.Rhistory
... then press C
The last two lines are to remove your personal R configuration files - since these should not be in the server.
Alternatively, to avoid the "svn delete", we can just selectively import the R source files we intend to store.

svn checkout --force svn://localhost/QuantAnalytics/myProj   ~/myProj
svn update
svn commit

- Close the Shell from Rstudio.
- Now back in the main Rstudio interface, Close the R project
- Reopen the R project. You will now find in the top-right Box, next to the "Workspace" and "History" tabs, there is a new "SVN" tab. This "SVN" will track any changes made to files which are under version control.



7. A few other tasks with SVN and R studio.

To UN-version control your local directory - the situation is directory A is in version control, and directory A is in your local workspace already. You wish to keep the contents in your workspace. You wish to break any links between your workspace and version control.
- go to directory A
- type: rm -r -f .svn

To delete any path or branch from version control. Warning: Deleting a path and files from version control may be deleting some files your colleagues are using. DO NOT do this until you check with them. Danger: this command may allow you to accidentally delete the wrong path in version control, ie. deleting your colleauges files. BE SURE what you are deleting.
- svn delete svn://localhost/QuantAnalytics/<your directory>

NEVER TYPE:
svn delete svn://localhost/QuantAnalytics





Services That Need To Be Restarted
===================================

Services are: Apache Rstudio    SVN
<ServiceNames>:     httpd        rstudio-server      svnserve
Runlevel default:       NA                  2345                NA
Runlevel default:      2345                2345                2345


Runlevel default is the out-of-box startup configuration before any changes have been made.
This is obtained from :  chkconfig --list

Do the following to register httpd and svnserve will start automatically
sudo chkconfig --level 2345 httpd on
sudo chkconfig --level 2345 svnserve on


Redhat Enterprise Linux runlevels are (according to /etc/inittab)
# Default runlevel. The runlevels used by RHS are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
#

Checking if daemon has started:
service rstudio-server status
chkconfig --list
sudo more /etc/inittab            - shows how the system is initialized
sudo ls /etc/rc.d/init.d/         - list of daemon files, used or not used in startup
ls -latrF /etc/rc.d/rc3.d/        - links to actual files in init.d, at this particular runlevel 3

How to Program

The following section discusses the use of Linux/Unix as a programming environment. Below is a list of some tools. More details are covered in the linked article.

Unix as IDE: Introduction
http://blog.sanctum.geek.nz/series/unix-as-ide/

File and project management   — ls, find, grep/ack, bash
Text editor and editing tools — vim, awk, sort, column
Compiler and/or interpreter   — gcc, perl
Build tools                   — make
Debugger                      — gdb, valgrind, ltrace, lsof, pmap
Version control               — diff, patch, svn, git


Some highly recognized books on programming:

The Little Schemer - Daniel P. Friedman
Mastering Algorithms with C - Kyle Loudon
C Programming Language (2nd Edition) - Brian W. Kernighan, Dennis M. Ritchie