Wednesday, October 07, 2020
Notes Crypto Coin (CRO)
Saturday, September 12, 2020
Notes LinkedIn
NotesLinkedIn
=================
How to Print Messages to PDF
How to Block someone without them knowing
How to Print Messages to PDF
================================
Printing to PDF is a very simple thing, or it should be. But by default, it seems very difficult to do to get the formatting just right. It is either cut off on the right hand (maybe we can change to Landscape). But then it only prints a little bit of the message on one page.
How to we print the many items of a very long message conversation? The trick is as follows.
- Use Chrome Browser
- Click Messaging on the top horizontal panel.
- The main screen has a left panel of various people, and the middle panel has the main message history of a chosen person.
- On the left panel, click on the person whose message you want to print, and check that the correct message history is on the main panel.
- For privacy, if you want to hide the people on the left panel before printing, go to the left panel, click on the triple dots on top, and select 'Manage Inbox Request'
- On the middle message panel, scroll up to see the start of the messages you want to print.
- Press Ctrl-P, choose Save as PDF.
- In the browser Print panel, Portrait was enough for my printing. But you can change to Landscape.
- In the browser Print panel, in Scale, choose a smaller number say 70% and try a few different numbers until the size and content of messages is OK for you.
- Then click Save. This will produce a PDF file.
- Open the PDF and print as usual.
The main step above is to choose the scaling to make sure it fits enough information of the messages you want to print.
How to Block someoone without them knowing
============================================
No problem, LinkedIn does the block discreetly. It says:
"The blocked member won't receive any notification of this action."
in here: https://www.linkedin.com/help/linkedin/answer/47081
The steps are very simple and listed on the link above.
Basically you have to go to that person's page, click on More, select Report/Block.
When a few options are presented, click on the option that says "Block [user name]". Then click Block.
Note: Blocking is different from Removing a Connection. With blocking it seems you are still connected but would not be able to see their profile, updates and cannot message them anymore.
To unblock is a bit different. Go back to your main page.
On top, click Me, click Settings and Privacy, click Block and Hiding, click Block and select Change. Then choose the person you want to block.
Whaling and Social Engineering by a LinkedIn Babe
On LinkedIn, I have been recently approached by a very professional seeming young lady. Have you had that experience? Then read on......
(Google Reviewers, please do read carefully before making any decision).
Firstly, I am a male, and I am susceptible to be very friendly with females who are friendly to me (I accept not all males feel the same). Female readers are very welcome to leave your comments, but please understand our weakness.
This article is about an experience which can potentially lead to severe security compromise for the victim, their relatives, friends, colleagues and employers. This article is about a potential account of a whaling or social engineering attempt.
There are various definitions of whaling or social engineering, but let's just use the definition below as an example (not that it is the 100% correct definition), so that we have a rough idea to proceed.
https://searchsecurity.techtarget.com/definition/whaling
"A whaling attack, also known as whaling phishing or a whaling phishing attack, is a specific type of phishing attack that targets high-profile employees, such as the CEO or CFO, in order to steal sensitive information from a company, as those that hold higher positions within the company typically have complete access to sensitive data. In many whaling phishing attacks, the attacker's goal is to manipulate the victim into authorizing high-value wire transfers to the attacker."
Guys, this paragraph is for you. It's not unusual I get unsolicited invites from LinkedIn. They include bearded guys whose name I cannot pronounce, I click Ignore. Nice looking gentlemen who are headhunters, maybe I accept or ignore. Various ladies who are headhunters, again I may accept or ignore. PhD or students of various types, I may accept if they have an interesting profile, otherwise I ignore. But I have never been invited by a babe-level (non-headhunter) who wanted to strike a conversation which is non-work / non-professional related - until now.....
Initial invitation -
So this under 30s, corporate entreprenuer and executive professional who appears to be running a company, decide to invite me in LinkedIn. Let's call her LX from here. To be specific her LinkedIn profile photo is not babe-level, but instead very professional and very expensive looking high class lady. So the invitation is unusual in the sense of it being rare (I don't get invitations like this) but not enough to be suspicious initially - maybe her work or project made her see something in my professional background that may be interesting - after all this is what LinkedIn is about, thank you. On a personal level, she is also the same ethnic background as myself, so in this case, that increased my interest a little.
Interaction -
The following is an account of the interaction and so far the interaction is still on going and there is no conclusion yet.
- After connection at LinkedIn, we started with well wishes and small conversations. Nothing unusual, but I'm quite excited to have this interaction.
- Not long after, LX admits she does not use LinkedIn much, and asks if I use WhatsApp or WeChat. Immediately I too wanted to move off from LinkedIn and continue our conversation privately in a non-professional platform for chats.
- On the Chat platform, the conversation continued. The conversations was very pleasant and non-committal. We started to talk about what each other is doing. From my personal view, I enjoyed this very much because it was refreshing to have a new friend who seemed to be very interested in my routine life. She was not nosy and not at all inquisitive of what I do. So no alarm bells there.
- I asked a few questions about her professional life, ie the business that she does. The questions were in the direction of trying to see if there is any synergies between my work and her work so there is more to talk about professionally, or help each other's business. Again this is quite standard in what LinkedIn is suppose to allow for.
- No business talk. However her response to work and business questions were polite and relevant, but usually ended quickly. For example, her company is multinational, and although they have offices in your country, she does not have direct dealings with your branch. This is quite plausible but adds or subtracts nothing in terms of evidence.
- How is your day? The conversation continues very politely and showing interest in daily life routine, such as how is your day, did you sleep well, have you eaten. These are all very nice and I'm very touched to have this level of interest from someone else.
- Niiice photos - Without making any requests, she voluntarily send me photos of herself doing sports (in tight sports gear, calm down you guys) and also showing her at dinner. The dinner picture of her is sensational - very photogenic. So of course I want to continue the interaction more.
- We talked about music, food, weekend activities. So very nice indeed. She does not seem to care about my family background, education background, social status. I have no pressure to try to impress her simply because she seems to have everything. Wow I really like to meet her. Even if for a purely platonic relationship (guys, please!), she is so refreshing to interact with.
- The summary of the interaction is so far we have both kept it very friendly and have not demanded anything from each other at all. Hence no pressure and no expectations. Other males in this position may have taken a different direction and I don't really know how she would respnd. In any case, at this point there is no cause for alarm based on the interaction in the chat room.
Let's now go to the other side of thought.....
Reality Check -
1. Who am I that such a perfect lady (looks, wealth, charm, humour) would pay attention to me? An old saying goes 'Look at your own cat-face (literal translation from another culture)', 'Look at your own mug-shot', 'Have you looked in the mirror lately'? This is one reality check that pulled me down to earth. Though not looking like Brad Pitt or George Clooney, etc, I do think I'm not bad looking at times. But with double chin, and an average look, really? would a pretty and wealthy and successful young lady choose me out of millions of nicer looking guys out there? (If you answer yes, you don't deserve sympathy if you are taken for a ride).
2. 'If it is too good to be true, then it is not true'.
Countless people with higher intelligence than myself have fallen victims to scams. I'm not qualified to give all the reasons why and how smart people get scammed, conned, tricked, but it happens all the time. To re-iterate many smart people fall for it. Simply put, when we see something is so good, and we really want it, then we will try to find all the reasons why this good thing can be true. It's not a matter of being optimistic that makes people vulnerable. Those who fall victim to scams, really do want the bait (money, pretty looking person, etc) - we all want something and sooner or later, there is scammer out there who would have the correct bait for us.
3. Use logic not our senses - A Beutifual Mind.
Men have often been accused of thinking, not with their brain, but with the organ below. If looking in the mirror does not bring a reality check (coz You're So Vain), then remember the movie A Beutiful Mind about the Noble Prize winning mathematician and schizophrenic John Nash. Spolier Alert: Since his university days, he was seeing an imaginary little child. Through his many years of struggle with schizophrenia including hospitalization and medication, he still believed the child was real because he saw with his own eyes, and hear with his own ears. This super briliant mathematician knew he had schizophrenia but cannot deny the existence of the little child. However, the Reality Check came one day, when he realised (using simple logic), that he knew this child (seeing and talking with her) for many decades of his life but she did not grow old. This is the reality check that even all his physical senses tell him it is real, simple logic about humans and ageing brought him out of his delusion and so he accepted the little girl cannot be real.
OK - let's put our suspiscious cap on for a moment. (Remember, nothing in the interaction so far seems like she wants anything or has anything to offer, except friendship. Neither of us have spoken anything other than what two very familiar friends would speak about.)
Profile:
- Profile photo included whole person, so her face is quite small in the picture.
- LinkedIn people also viewed other ladies with profiles which are very suggestive, yet with high corporate titles.
- She has a 500+ followers. Nothing wrong with that by itself.
- She has a very important, high position title, but somehow the wording is not quite right.
- Education and work experience profile is quite short. For a yound person, this is understandable. Two universities were listed. Three work experiences listed, the first is an internship. The third is the current very high and important sounding role.
None of these stand out too much by itself, but ordinarily I would not have accepted the Connect invitation with someone with a very brief education and work experience profile. So my confession is that I accepted because she was an interesting lady.
Suspicions - Putting on the tin-foil hat.
The conversations in the chat does not indicate she wanted anything from me, neither information nor resources. The LinkedIn profile described above is not unusual for each individual item, but collectively, they are starting to ring the alarm bells. From there, trying much harder to look through all the evidence, here is a list of suspicious items.
- In the interaction / chats, when I tried to talk about her business to stimulate her interest or passion in her work or business, her answers very soon changed the topic. It was not avoiding the question, but the answers were very plausible, but like 'now let's talk about something more interesting in your personal life'
- She does not ask me about my work, except simply 'what I do at work'. There is no attempt to extract any kind of work related information at all. In fact this almost lack of interest about my professional experience, considering we met at LinkedIn is a bit unsual I thought.
- She has so far send two photos. The photo in a sporting activity shows her in a tight exercise suit - revealing a nice body (sorry, I have say it as it attracted my attention). The one at dinner was very proper looking, showing such a cute and sweet face. But comparing these two photos with her LinkedIn profile photo and the chat app profile photo, I cannot say for sure that it is or it is not the same person. Each photo was taken either at a different angle, with different hair style and different pose, basically enough differences that it could still be the same person.
- Photos - another thing about the photos is that they are not selfies. So she always has someone with her, taking these nice photos of her, whether in an action during sports or at dinner for one, yet looking nice and framed properly in the photos. Who took these photos? Was she with someone? If there is someone, even a friend, why would she be chatting with me half way around the world instead of giving more attention to that person who is with her?
- Looking more closely at the education level in combination with the work experience. She attended two universities consecutively from 2011-2017. A brief internship was done. Then it appears her first job after the internship is CEO of a company (4 years ago until now) and and second job at the executive level from (3 years ago until now). So this is what I meant by a seemingly successful or wealthy young lady. Can this be true? Yes since her country is a place where rule-of-law is not always true, and important people (e.g. her father) has significant advantages. Or it could be a not-very-good fake profile.
- Looking at her LinkedIn connections, it is almost all males out of the first 8 pages with only one female who is another high-powered successful lady. All the males are CXO, C-level executives, directors, Heads of xyz, business leaders etc. Not even head-hunters has such high powered connections. Could all these male high-flying individuals be potential victims?
Why me? What for?
Although some alarm bells may start to tingle, most likely we may not know the reason until it is too late. Here are just a few very general reasons and there could be more.
https://www.smh.com.au/lifestyle/life-and-relationships/don-t-know-why-i-sent-more-sally-lost-her-entire-savings-to-a-love-scam-20200909-p55ttb.html
1. The article above is about old-fashioned scamminng involving romantic love, but the technology could be modern, like chat or video conferencing. The scammers are patient and build trust and love on the victims, then the objective of the scam is old-fashioned money.
2. For more modern whaling, the objective would be to target high level corporate leaders, or someone influential enough in an organisation, such as an IT Administrator. Then if the scammer can infect the devices of the victim, potentially everyone else in the victim's computer network (ie the office) could be vulnerable.
3. Social engineering - this may combined the old-fashioned charm and get-to-know-the-person-trick, then with the similar objective as whaling, by having the goal of being able to access information or computer systems of more high value targets in the organisation of the victim.
I don't think I can fall in the romantic scam trick, but that is in fact what I could fall for. I don't have a high title in my company nor important role, but that does not mean I cannot be used to get to another person through me. Just because I don't know why, does not make me any less valuable to a scammer.
4. Here is another potential reason (I won't comment further)
https://www.asianage.com/world/americas/310818/china-using-fake-linkedin-accounts-to-recruit-americans-us-top-spy-catcher.html
Checks
The suspicions above may or may not reflect the truth - and I still cannot be sure at this point.
However, a few things can be checked.
- Contact University? Alumni Lookup service?
- Check with a few others she has been in contact with.
- Check by calling the work company, maybe not directly, but approach with tact.
- Image check in the internet.
- Common scammer fake profiles search
Update:
I just verified using reverse image search (https://tineye.com/search/), that the chat app profile photo is taken from a young lady at a tourist website. The website has a lot of photos of the same lady and they are clearly recognizable as the same person. However, LX's various photo seem difficult to identify as the same person.
Sunday, August 23, 2020
Offline Browser, Proxy Server, Cache for viewing web pages without the Internet
The following is a work in progress to find a way to view web pages without the internet.
How to Enable Offline Browsing in Chrome - an experimental feature within Chrome and like browsers.
https://www.hongkiat.com/blog/windows-offline-browsers/ - a good list of offline browsers
https://proxy-offline-browser.com/professional/Installation.html - MM3 Web Assistant. This functions as a proxy and automatically saves the webpages that have been visited. However the free version does not handle HTTPS. The benefit is that users do not need to specify which sites to save specifically.
HTTrack Website Copier - seems to be the best so far. Will test soon.
WebCopy - similar to HTTrack but not JavaScript....
Wiki for Offline browsing
- Download the linux version: http://xowa.org/home/wiki/Help/Download_XOWA.html
- On Linux, open a terminal and run sh /home/Your_Name/xowa/xowa_linux.sh
- Note that the shell script is a wrapper for java -Xmx256m -jar /home/Your_Name/xowa/xowa_linux.jar
- Path of jar file /home/Your_Name/xowa/xowa_linux.jar
Friday, August 21, 2020
Notes Video Editing
https://au.oberlo.com/blog/best-free-video-editing-software
I'll add comments to help narrow down even further for myself basically - but others may find it helpful too.
Blender - originally for 3D animation. I've also heard about its not-so-easy to learn aspect, though it has lots of features. But I won't be trying this.
Lightworks - the Free version has a limit of 720p. I've used another converter to convert to 1280x720 and compare with the original and very common 1920x1080 size. The 720 one is noticeably fuzzier. So I won't be testing this further.
Shortcut -
While the cut function seems OK, the Export is difficult to configure. Pressing the Export button does nothing. In one case when it did export, the file is more than twice the original.
- not really that straight-forward to use without using any help
- Export to a lot of formats, but for some reason, the output is 10 times larger than the original. Why it doesn't use the exact compression level of the input?
- BEST so far
- Small footprint, about 80MB fully installed (according to the installation dialog)
- Best Feature - COPY - this option in the video encoder format allows it to use exactly the same video format as the input. So if there are no edits being made, the new output is exactly the same size. This is very useful in comparison with other software above which, in most cases, result in an output which is larger than the input video file.
- limited to 7 minutes clip for free version
Sunday, July 26, 2020
Tensorflow Deployment Requirements for Machine Learning / Neural Networks
Monday, June 22, 2020
Unboxing - AIMOS HDMI KVM 2-port Switch
 |
| Chinese |
 |
| Add caption |
Tuesday, April 28, 2020
Notes q
An Overview of the q language
You Tube - Introduction to KDB+ and q
========================================
Tutorial: https://www.youtube.com/watch?v=8eoysfqO3UY&list=PLypX5sYuDqvrwBD2EMWadIMiTqJZmVsqm
The following is a transcribe of the details in the above introduction course. It is meant as rough notes to capture the syntax and used as a reference, and may not give good worded descriptions.
#lesson 1
NO: control flow, loop, if-else, threads, shared globals, objects, inheritance
interpretive language, dynamically type, vector programming, functional programming, list, dictionaries, tables (1st class items, based on list, dicts)
#lesson 2
function has TYPE
42 -> 64 bit
42i -> 32 bit
4%2 -> % is division
2f -> float
#lesson 3
1 2 3+10 20 30 -> vecotr operation, results is 11 22 33, spaces are used to separate list items
count 1.0 2.0 3.0 -> 3 is the result
1+10 20 30 -> 11 21 31 scalar(atom) + vector
# All operations have precedence, operations done right to left
2*3+4 -> (3+4) =7 then *2 =14
til 100 -> generates 01 2 ... up to 100 numbers
count til 100 -> 100
101+2*til 1000 -> generate (0 .. 999), then x 2, then add 100 to every item. Result: 101 103 105 ....
#lesson 4
0b -> false
1b -> true
101b -> list of booleans true, false, true, NOT separated by space, and NOT a bitmask representation
42=6*7 -> 1b (true)
1 2 3=10 2 30 -> 010b vector results of false, true, false
2018.01.01 -> actually count of days since millenium (yyyy.mm.dd)
2000.01.01=0 -> 1b True
1999.12.31=-1 -> 1b True
2000.02.01-2000.01.01 -> 31i (31 days, in INTEGER 'i') date arithmetic
2000.01.01+til 31 -> make 31 integers then add to 1st Jan 2000 -> gives the dates of that month. List-Date arithmetic
2000.01m -> month
2000.01m=0 -> 1b Comparing the 1st month since millennium
2000.02m=1 -> 1b Comparing the 2nd month since millennium
2000.01.01=2000.01m Comparing 1st day ever, to 1st month ever, q assumes the month is 1st day of month when comparing to date, hence true.
#lesson 5 - Casting
`long$1.0 -> cast to long
`float$1 -> cast to float
`boolean$1 -> cast to boolean
`date$31 -> cast to date format, ie 31 days since millennium is 2000.02.01
2000.01m+til 12 -> list of 12 months 2000.01 2000.02 2000.03 ...
15+2000.01m+til 3 -> 2001.04 2001.05 2001.06m 1st 3 months, after 15 months from the millenium
# lesson 6 - Operations on List
Declarative programming - what we want in q. Other languages are imperative, ie write loops to tell how to compute.
0 +/ 10 20 30 40 50 Initialise to zero then add the items on the list. The +/ symbol called 'over' in q, 'fold' in functional language.
(+/) 10 20 30 40 50 Same as above - add across a list. Does not need to be initialised.
'/' The slash is ALSO behaving like an iterator
sum 10 20 30 40 50 -> same as above
(*/) 1+til 5 Does Factorial(5)
4|5 -> 5 returns larger number
10&2 -> 2 returns smaller number
0b|1b -> 1b Larger operator for Boolean is equivalent to 'OR' operator
0b&1b -> 0b Smaller operator for Boolean is equivalent to 'AND' operator
(|/) 10 30 20 -> 30
max 10 30 20 -> 30
(&/) 10 30 20 -> 10
min 10 30 20 -> 10
(+\) 10 20 30 40 50 -> 10 30 60 100 150. Backslash means create a CUMULATIVE / RUNNING list. also called 'sums'
(&\) 40 20 30 10 50 -> 40 20 20 10 10. Running minimum. Also called 'mins'. Similarly with 'maxs'
# Lesson 7 - Functions
{[x]x*x} -> [x] is the parameter
{[x]x*x}5 -> substitutes 5 into x. Result is 5*5. Can also use {[x]x*x}[5]
{x*x}5 -> implicit parameter, don't need to declare.
# Lesson 8 - Newton Raphson and Fibonacci Sequence
Say Function is f(x) 2-x^2. So f'(x) = -2x. NewtRaph algo is xn = xn - f(x)/f'(x)
{[xn]xn+(2-xn*xn)%2*xn}/[1.0] -> 1.414214 .... result from Newton Raphson using 1.0 as initial point, being put into the function on the left iteratively.
{[xn]xn+(2-xn*xn)%2*xn}\[1.0] -> 1 1.5 1.4..... shows the intermediate results from Newton Raphson
\P 16 -> shows 16 decimals. iteration stops until diff is 1e-14
2#10 20 30 40 50 -> 10 20 '#' retrieves the integers from the left of list
-2#10 20 30 40 50 -> 40 50 '#' retrieves the integers from the right of list
10 20, 100 200 300 -> 10 20 100 200 300 joins lists
{x, sum -2#x}/[10;1 1] -> returns the Fibonacci Sequence. Iterate 10 times, starting with [1 1]
# Lesson 9 - Function, Variables
deltas 110 120 130 -> 110 10 10 takes differences between two numbers. The first result assumes difference with zero.
deltas sums 110 120 130 -> is the same as sums deltas 110 120 130 -> result is 110 120 130. Invariance in q.
a:42 -> assign value of 42 to variable a.
buys:2 1 4 3 5 4 ->
sell:12 -> want to sell 12 units from the various buys
sums buys -> 2 3 7 10 15 19
sell&sums buys -> 2 3 7 10 12 12 - creates a list of minimum between two lists.
deltas sell&sums buys -> 1 2 4 3 2 0 - list of what how should buy
# Lesson 10 - Tables
Table = Collection of columns in q. Operations are column based, hence vector operations.
dates:2018.01.01+10000000?31 -> list of 10 million numbers between 0(including BUT not stated) and 31(stated, but EXCLUSIVE)
times:10000000?24:00:00.0000 -> 10 million random nums, between 0 (including, BUT not stated) and midnight (EXCLUSIVE)
qtys:100*1+10000000?100 -> 10 mil random numbes from 0 to 100, then +1, then * 100
ixs:10000000?3 -> 10 mil rand num, from 0 1 2. store random numbers in ixs
10 20 30 ixs -> 10 10 30 20 30 20 ...... Since ixs has 3 unique values, so the 10 20 30 will apply to 10 20 30, MAPPED into the 10 million entries of ixs.
syms:`appl`amzn`googl ixs -> `googl`aapl`amzn`googl`googl .... applies symbols to list, store in syms. The 3 stock quotes is MAPPED to the 3 unique values of ixs for all 10 million values of ixs
172.0 1189.0 1073.0 ixs -> maps the 3 unique values into the 10million numbers of ixs
pxs:(1+10000000?.03) 172.0 1189.0 1073.0 ixs -> adds some random noise to the 10 million stock prices.
t:([] date:dates;time:times;sym:syms;qty:qtys;px:pxs) -> creates a table, with specified column names, and columns of data
t:`date`time xasc t -> sort ascending time, then date
5#t -> returns top(head) 5 rows
# Lesson 11 - qSQL
select date,time,qty,px from t where sym=`aapl
\t select date,time,qty,px from t where sym=`aapl -> returns time in milliseconds
first 10 20 30 40 50 -> first item on list
last 10 20 30 40 50 -> last item on list
select open:first px,close:last px by date,time from t where sym=`aapl
#Lesson 12 - Complex Queries(*)
4 3 2 1 wavg 10 20 30 40 -> 20f calculate weighted average
5 xbar 0 1 2 3 4 5 10 11 21 -> 0 0 0 0 0 5 10 10 20 returns bucketed result. Useful for time domain values that needs bucketing
select max px - mins px from t where sym=`appl -> is the Max idealised profit (ie buy at the lowest price of the day and sell at the highest price of the day. The function is the same as select (max (px - (mins px from t where sym=`appl) )). The mins is a running cumulative vector of the minimum, while the max just returns the single value.
#Lesson 13 - Interprocess Communication(*)
Server and Client - ie running two different q.
On Server
\p 4242 Starts the server on port 4242
On Client
h:hopen`::4242 Where machine name is omitted between two colons, since it is on the same machine. h is the handler
h "6*7" Do compute on remote server
sq:{x*x} Send function to remote server
h (sq; 5) Execute on remote server
DON'T DO ABOVE since open ports are dangerous
On Server
cub3:{0N!x*x*x} 0N! displays computation on local server first, before shifting to client
On Client
h (`cub3; 5) Safer this way, CALL the name of the function. First Argument is function name, the rest are arguments
#Lesson 14 - Callbacks(*)
Previously the client server were synchronous, meaning the parties wait for each other after sending a message.
For Asynchronous, when message is sent, the sender does not need to wait.
The handler h is an 'integers'.
On Client
h -> 3i is the integer 3. this is a function handler
(neg h) (`cub3;5) -> h is positive, neg h is negative. So "(neg h) (`cub3; 5)" is an ASYNCHRONOUS call.
Callback: Without callback, an asynchronous function would send message to server, then receives no response. With callback provided on the server, the server will send the result back whenever it is ready.
Client uses 'h' as its handler to call the server
Server uses '.z.w' as its handler of the function send from the client. .z.w returns 6i
On Client (FINAL)
continue:{0N!x} -> displays x over here
(neg h) (`worker; 5; `continue) -> (neg h) for asynchronous, call process 'worker' on server, with argument '5', use the callback label 'continue'. First Argument `worker is function name, the rest are arguments [5; `continue]
On Server (FINAL)
cub3:{0N!.z.w; x*x*x}
worker:{[arg; callback] r:cub3 arg; (neg .z.w) (callback; r) -> 'worker' is process called by client, defined here on server. [arg; callback] are the parameters passed from client. 'callback' is the dummy name corresponding to 'continue'. Runs cub3 on arg, store result in r. Then (neg .z.w) triggers the callback to client asynchronously. Thirdly execute the 'callback' (which is 'continue') with result r.
# Lesson 15 - I/O
Strings - abit different than other languages. Strings are a list of characters, not a primitive type.
"jab" is a list
`jab is a variable name
Binary files - operations have 1 in their names
Text files - operations have 0 in their names
File records are list of strings to q. Which means they are a list of list of characters to q.
("So long"; "and thanks"; "for all the dish") - a list of list of chars
count ("So long"; "and thanks"; "for all the dish") -> three list items
`:/data/solong.txt 0: ("So long"; "and thanks"; "for all the dish") -> `:/data/solong.txt returns this file handler as acknowledgment
' error messages are 'Straight ticks'
` File handlers are 'back ticks'
`: is the file handler or names a file.
/data/solong.txt is the local path
0: is the name of the WRITE to text file function
read0 is the name of the READ from text file operation
read0 `:/data/solong.txt -> prints out the lines of the file
`:/data/answer.txt 0: txt,txt:read0 `:/data/solong.txt -> `:/data/answer.txt returns this file handler as acknowledgment
Date - starts from the millennum
Vector Column operations - not Rows
select from 10million rows in 150ms
Single line of code for even for non-trivial functions
select max px - mins px from t where sym=`appl Algorithm for Idealised Profit in Trading (sym is symbol column, px is price column of table t )
{[xn]xn+(2-xn*xn)%2*xn}\[1.0] Algorithm for Newton Raphson with initial point as 1.0
{x, sum -2#x}/[10;1 1] Algorithm for Fibonacci Sequence, initial points 1 1 , for 10 iterations
An Overview of the q language
================================
'q' is a proprietary language build on top of an existing 'k' language (developed by ex-Morgan Stanley computer scientist Arthur Whitney) and the kdb+ datatbase (The 'q' language / kdb+ is free for non-commercial use). They are all proprietary products of Kx Systems and it is perhaps more widely known (if not the de facto) in the finance and investment banking industry more than any other industries. As such, it is aimed at time-series based data, large volumes and high transfer rates. Very complex applications can be written in the 'q' language with an extremely small footprint. A large number of Altair's real time visualisation software, Panopticon, connect to kdb+ databases and utilises q.
A recent exploration into the depths of the q language reveals a very sophisticated and expressive computational programming language. It is both like and unlike various programming languages. Perhaps the first point is the vector programming underlying its core, yet this language may not be known widely in the HPC community (or perhaps I have not looked wide enough while I was in HPC, it happens). Vector supercomputers once ruled the HPC world, but when they approached extinction, the vector processing capability were re-incarnated into commodity CPUs and is now powering our PCs (since the days of Intel MMX, SSE, etc...). And it appears that around that similar time shortly after the Millennium, 'q' was born independently of Vector Supercomputers or vectorised CPU instructions, but rather tracing its origin back to the 60s from APL (A Programming Language), thus having both vector and functional programming influence.
For reference, a free book is available at: https://code.kx.com/q4m3/0_Overview/
And a Tutorial series is at: https://www.youtube.com/playlist?list=PLypX5sYuDqvrwBD2EMWadIMiTqJZmVsqm
For those coming from scientific, computational, HPC background, thinking in terms of vectors will ease our understanding to 'q'. For the general programmers who are familiar with C, Java, Python, and the rest, the good advice from the YouTube link would be to throw away your understanding of control flow, loop, if-else, threads, shared globals, objects, inheritance. The syntax itself is short, succinct and raw, yet expresses beautifully the operations it performs. The following will describe some of the noteworthy things about 'q' which are different than the current popular programming languages.
'q' is similar to many scripting language, ie interpretative (no compilation), dynamically type (no pre-declaration of data type). The two fundamental data structures are lists and dictionary, again no surprises here. However it is the vector operations that are perhaps new to most modern day programmers. For example a vector 1 2 3 can be added directly to another vector 10 20 30. Yet these vector expressions should be reminiscent of those in Matlab and Fortran, but the similarities don't extend much further. The syntax is perhaps quite different (though perhaps not too dissimilar to other functional programming languages). The vector (list) items are delimited by spaces (not commas). The mathematical operators has equal precedence (no BODMAS rule) and the order is designed so that the operations are performed in the right-to-left order. For example: 2*3+4 is actually 2*7, which is 14. This right-to-left and single line way of writing code is perhaps the biggest change a programmer of other languages need to overcome.
Control flow and loops goes out the window because an operation acts on all the items in the vector. There is no point in having an explicit index to loop over. As such, an operation over 10 or 10 million items can be written in one line. In fact, entire functions or methods are written in one line, as shown in examples later. In terms of data types, there are the familiar char, boolean, integer, float and so on. Strings are like in Fortran or old C, a string in q is actually a character of arrays. The really most interesting data type I found is the Date representation. Instead of some arbitrary representation having an initial value at 1970 or 1900; q's date are actually integers starting from 1 Jan 2000. This means date computations benefit from the speed of raw integer operations, as well as being able to perform fancy date arithmetic.
List operations are powerful, since this is one of the core data structures of q. List, or vectors can be operated with another vector or scalar, not dissimilar at all to their mathematical counterparts. In addition to aggregate operations like 'max', 'min', 'sum', there are more interesting versions 'maxs', 'mins', 'sums' which actually returns a cumulative vector of the aggregation process. The highlight of the 'q' language of its expressiveness and power would be its function definitions. Instead of describing, ponder upon the following one-liner, and consider how such functions would need to be written in Python, Java, C, etc:
Algorithm for Newton Raphson with initial point as 1.0
{[xn]xn+(2-xn*xn)%2*xn}\[1.0]
Algorithm for Fibonacci Sequence, inital points 1 1 , for 10 iterations
{x, sum -2#x}[10;1 1]
Algorithm for Idealised Profit in Trading (sym is symbol column, px is price column of table t )
select max px - mins px from t where sym=`appl
As for Data Analytics and complex data processing, there are tables, qSQL and a few more useful functions. Tables are collection of columns, like Fortran/Matlab but opposite to C-based languages. Columns in tables generally mean they are of the same data type that means they are easily stored as vectors. For q, this means performing operations on large datasets are extremely fast. Combining the single line functional syntax with vector based operations, very complex yet efficient programs can be written in a terse manner. The qSQL syntax enables very SQL-like expressions to be written. This may be a comfort for SQL programmers, but it still pays to understand the underlying concepts of q, since there are fundamental differences that can lead to incorrect expectations. Native functions like 'xbar' that creates bucketing are extremely powerful when applied to time-series based analysis. In an example of 10million rows, a certain select operation took 150ms.
Most programming languages have the capability of File I/O. q also has File I/O for both text and binary files (again there is the similarity with Fortran's binary files capability, rather than the streams based files I/O of other languages). Not surprisingly by now, q's way or reading and writing files are one-liners. There is no need to open or close a file or the many other lines of instructions needed by other languages. Besides File I/O, q is also capable of Interprocess Communications and Asynchronous processes. Together, these capabilities enable q to write client-server applications without using any other external tools.
From this brief high level exploration, q seems to be an interesting, powerful and useful programmig language. It has the expressiveness of functional programming, the power of vector processing and the applicability to real high volume data computational needs. The fact that it is born within the Financial Trading sector by no means exclude it from being used in other industries. Whereever there is large numerical datasets that requires computation, q may be a good choice. These other industries that may benefit include: data science, scientific research, defence, telecommunications, etc. In terms of providing real-time visualisations of the large and fast datasets, Altair's Panopticon with kdb+/q would make a formidable combination.
Wednesday, April 08, 2020
Notes Kafka Viewer
https://github.com/Landoop/kafka-topics-ui
https://github.com/confluentinc/kafka-rest
1. Kafka Web Console https://github.com/claudemamo/kafka-web-console
Last Update: ***This project is no longer supported. Please consider Kafka Manager instead.
https://github.com/yahoo/kafka-manager
- a Java web application for monitoring Apache Kafka. With a modern web browser, you can view from the console:
- requires a relational database. By default, the server connects to an embedded H2 database and no database installation or configuration is needed. Consult Play!'s documentation to specify a database for the console. The following databases are supported:
2. Kafka Manager https://github.com/yahoo/kafka-manager
Last Update: A month ago
A tool for managing Apache Kafka.
Starts as a service (but how to run??)
It supports the following :
- Manage multiple clusters
- Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
- Run preferred replica election
- Generate partition assignments with option to select brokers to use
- Run reassignment of partition (based on generated assignments)
- Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
- Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
- Topic list now indicates topics marked for deletion (only supported on 0.8.2+)
-Batch generate partition assignments for multiple topics with option to select brokers to use
- Batch run reassignment of partition for multiple topics
- Add partitions to existing topic
- Update config for existing topic
- Optionally enable JMX polling for broker level and topic level metrics.
- Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.
3. Landoop LENSES Box (as oppose to Enterprise) - http://www.landoop.com/downloads/
Free Version:
All in one docker*
Single broker up to 25M messages
***Lenses with all your kafka setup with extra features, data generators and examples in a single docker environment ideal for development straight in your laptop!
PERHAPS NOT COMPATIBLE SINCE WE'VE ALREADY GOT KAFKA SETUP
4. Kafka Tool - http://www.kafkatool.com/
- GUI application for managing and using Apache Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators. Some of the key features include
-Quickly view all your Kafka clusters, including their brokers, topics and consumers
-View contents of messages in your partitions and add new messages
-View offsets of the Kafka consumers, including Apache Storm Kafka spout consumers
-Show JSON and XML messages in a pretty-printed format
-Add and drop topics plus other management features
-Save individual messages from your Kafka partitions to local hard drive
-Write your own plugins that allow you to view custom data formats
-Kafka Tool runs on Windows, Linux and Mac OS
Kafka Tool is free for personal use only. Any non-personal use, including commercial, educational and non-profit work is not permitted without purchasing a license. Non-personal use is allowed for evaluation purposes for 30 days following the download of Kafka Tool, after which you must purchase a valid license or remove the software.
5. Kafdrop: An Open Source Kafka UI
This tool is definitely still in it’s beginning stages.
Kafdrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a light weight application that runs on Spring Boot and requires very little configuration.
- Easy Stand Alone Run, just specify host:port of zookeeper
- Java8 based
6. kafka-topics https://github.com/Landoop/kafka-topics-ui
Browse Kafka topics and understand what's happening on your cluster. Find topics / view topic metadata / browse topic data (kafka messages) / view topic configuration / download data. This is a web tool for the confluentinc/kafka-rest proxy.
docker pull landoop/kafka-topidcs-ui
docker run --rm -it -p 8000:8000 \
-e "KAFKA_REST_PROXY_URL=http://kafka-rest-proxy-host:port" \
-e "PROXY=true" \
landoop/kafka-topics-ui
7. https://github.com/jchen86/kafka-topic-viewer
need GULP
npm install gulp-build
npm: relocation error: npm: symbol SSL_set_cert_cb, version libssl.so.10 not defined in file libssl.so.10 with link time reference
[root@ZenOSSPrimaryS client]# sdk
-bash: sdk: command not found
8. https://github.com/gameontext/kafkaviewer
need GRADLE -> SDKMAN ....
9. https://github.com/claudemamo/kafka-web-console
Need Play Framework
10. I've run across https://github.com/SourceLabOrg/kafka-webview which offers a web socket "live" mode which I imagine would provide a high-level of responsiveness. Docker image available and looks decent (code-wide and GUI-wide based on screen-shots). Just flagging this as another possibility.
11. https://www.conduktor.io/download
*** Best so far
This is newish, as of Apr 2020 - but it looks to be working well in my quick tests.
It was installed on a Win7 machine and connecting to another machine running Kafka on Linux.
Many big positives going for this.
- Both Linux and Windows version available.
- For Windows, there is MSI, AND also simply a JAR version. With the JAR version, it does not need to be installed at all, just double-click to run the JAR file, provided Java is installed on the Windows system.
- Very easy to use and simple interface.
- Need to register, then login, but totally free to use.
- Some other types of Kafka Viewer are either Linux or require Docker. This is the best so far, no Docker and works on Windows.
Monday, March 16, 2020
Novel Coronavirus Covid-19 Data Analytics links
https://www.datanami.com/2020/03/12/tracking-the-spread-of-coronavirus-with-graph-databases/
Describes using Graph Databases to track infections.
https://www.health.nsw.gov.au/Infectious/diseases/Pages/coronavirus-flights.aspx
Official government site to track flights into NSW
https://ncov2019.live/data
Very good collection of data.
https://www.worldometers.info/coronavirus/
Another site with data.
https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset
Kaggle dataset
https://pages.semanticscholar.org/coronavirus-research
Links to many free research papers on Covid-19
https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6
John Hopkins dataset with GCP
https://cloud.google.com/blog/products/data-analytics/free-public-datasets-for-covid19?_lrsc=efc3f9b6-6095-4fe3-bd6f-c169a6131b21&utm_source=elevate&utm_medium=social&utm_campaign=elevate_TwG&utm_term=elevate_TwG_SEID:304699_BID:928653_TID:2749_SHID:18779651
https://www.cdc.gov/coronavirus/2019-ncov/about/index.html
Friday, January 10, 2020
Data Science - Machine Learning Datasets for the public
Kaggle
https://www.kaggle.com/
UCI Machine Learning Dataset
https://archive.ics.uci.edu/ml/datasets.php
Openml.org
Melbourne City Council
Open Data Platform
https://data.melbourne.vic.gov.au/
Centre of Land Use and Employment
https://data.melbourne.vic.gov.au/clue
https://www.melbourne.vic.gov.au/about-melbourne/research-and-statistics/city-economy/census-land-use-employment/Pages/clue-data-reports.aspx
Australian Bureau of Statistics - may need to search a bit
https://www.abs.gov.au
or a specific example:
https://www.abs.gov.au/ausstats/abs@.nsf/Latestproducts/5512.0Main%20Features42017-18?opendocument&tabname=Summary&prodno=5512.0&issue=2017-18&num=&view=
The U.N.
https://www.un.org/en/databases/
a specific example:
https://unstats.un.org/unsd/demographic/products/default.htm
Stock Trading Data
https://www.worldtradingdata.com/
https://finance.yahoo.com/
http://www.eoddata.com/register.aspx
10 New Ways to Download Historical Stock Quotes for Free
https://www.quantshare.com/sa-620-10-new-ways-to-download-historical-stock-quotes-for-free
Specifically, Technical Analysis indicators are available in the Python library TA-Lib. A description of how this can be used is found in:
https://towardsdatascience.com/trading-strategy-technical-analysis-with-python-ta-lib-3ce9d6ce5614
Thursday, January 09, 2020
Chess Software
The first step was searching for "free chess database" and found below.
https://www.chess.com/forum/view/general/world-largest-offline-chess-database-for-free
On that site, codekiddy has collected over 30million games over 500 years.
So how do we use his database? He says:
"I've been using Scid vs PC program to manage database and therefore the database is in scid format, meaning you can download it and do with it what ever you want, such as importing more games."
So the second search is for SCID which led to:
https://en.wikipedia.org/wiki/Shane%27s_Chess_Information_Database
http://scid.sourceforge.net/
That is actually a software that can potentially use codekiddy's 500 year cess database.
But wait, codekiddy actually says "Scid vs PC" as the name of the program. This program also appears in the wiki page reference, leading next to.....
... the third item "Scid vs PC".....
http://scidvspc.sourceforge.net/
Now to try out these stuff......
Wednesday, January 01, 2020
Cloud Compute Free Tiers - AWS, Azure, Google GCP, AliCloud, Oracle
What is important when selecting a "FREE TIER" in Cloud Compute resources for 'me':
- Long time frame - anything less than 12 months I would not waste time considering.
- Ease of sign-up - eg don't ask for your credit card
- Amount of Compute Resources. Yes I put this at 3rd rather than 1st place, because I expect most providers to give very tiny amounts of resources anyway.
I also provide the direct link to the exact specifcations, which often is not very obvious and hidden away from the main marketing page.
The details below are at the time of writing Jan 2020.
Google Cloud Platform (GCP) offers the f1.micro instance.
https://cloud.google.com/compute/docs/machine-types
For 12 months
"Micro machine type with 0.2 vCPU and 0.6 GB of memory, backed by a shared physical core."
Not even 1 whole core.
Alibabacloud offers the t5-lc1m1.small
https://www.alibabacloud.com/product/ecs-t5
12 months
1vcpu, 10% baseline performance, 1GiB
Microsoft Azure offers the Standard B1s
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/b-series-burstable
12 months
1vcpu, 1GiB, 10% baseline performance,
Amazon AWS offers the t3.micro instance
https://aws.amazon.com/ec2/instance-types/t3/
12 months
2 vcpus, 1.0GiB, 10% baseline performance
The oldest cloud player here, yet more generous than the above so far.....
Oracle Cloud offers ??? instance
https://www.oracle.com/au/cloud/compute/pricing.html
ALWAYS Free
The specs list in the link, cannot be used because it is not clear which VM instance is provided.
But on the main page, it says:
Databases 2 databases total, each with 1 OCPU and 20 GB storage.
Compute: 2 virtual machines with 1/8 OCPU and 1 GB memory each.
Storage 2 Block Volumes, 100 GB total. 10 GB Object Storage. 10 GB Archive Storage.
But on the link above, the smallest VM is:
VM.Standard.E2.1 with 1OCPU, 8GB mem, 1PB block volumes.
That was a terrible experience! Tried registering for the free cloud tier, and used my credit card. But the application got Rejected, because the Credit Card I used was a prepaid VISA credit card. Even though it is supposed to be Free - they still want to keep your main credit card number, on servers that can never be guaranteed to be secured.
Sorry - No winners now..... still looking for a decent free tier cloud.
Disclaimer: Again, this is my personal note and opinion. This is NOT any kind of recommendation.
Posts
