Move/Mirror/Migrate all your bitbucket repos to github

1.Add ssh keys to your bitbucket and github accounts

https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html#SetupanSSHkey-ssh2

https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/

2.Generate github token for reading repo info and read/write permission for creating repos (you can give all permission except delete )

https://github.com/blog/1509-personal-api-tokens

3.Use this script ‘bb2gh.py’ @ https://github.com/nmi/bb2gh or (https://github.com/kanihal/bb2gh)to mirror the repos from bitbucket to github.

 

Advertisements

Quotes

all models are wrong but some are useful – https://en.wikipedia.org/wiki/All_models_are_wrong

archive.org tools and stuff

http://machawk1.github.io/wail/

Upload a site to archive.org

http://archiveteam.org/index.php?title=Internet_Archive#Uploading_to_archive.org

1.crawl save each webpage seperately

 wget http://web.archive.org/save/

2.https://github.com/n0tan3rd/wail

make warc file and upload to archive.org

http://archiveteam.org/index.php?title=Frequently_Asked_Questions

I uploaded a WARC file but why doesn’t it show up in Wayback Machine?

To ensure content integrity, items with WARC files must have the mediatype set to “web” and be under the Archive Team collection in order for it to be ingested by the Wayback Machine.

 


 

Locally browsing with warc file –

https://github.com/webrecorder/webrecorderplayer-electron

 

Tools 

https://webrecorder.io/

http://rhizome.org/software/

https://ws-dl.cs.odu.edu/Main/Software

http://archive.org/help/abouts3.txt – s3 api key -> http://archive.org/account/s3.php


 

Download a site/page from archive.org for given time

modify and use this script -> https://gist.github.com/lazanet/872f88c9874e4a7a78fd

 

https://gareth.halfacree.co.uk/2013/04/bulk-downloading-collections-from-archive-org

 

Drive sync options on ubuntu

1. insync is the best option that is available for ubuntu (if you are willing to pay for it)

2. odrive (you can mount only one account at a time)

Download and install

https://docs.odrive.com/docs/odrive-sync-agent

od="$HOME/.odrive-agent/bin" && curl -L "http://dl.odrive.com/odrive-py" --create-dirs -o "$od/odrive.py" && curl -L "http://dl.odrive.com/odriveagent-lnx-64" | tar -xvzf- -C "$od/" && curl -L "http://dl.odrive.com/odrivecli-lnx-64" | tar -xvzf- -C "$od/"

Add theses to end of your .bashrc file

export PATH="/home/jkl/.odrive-agent/bin:$PATH"

alias startodrive= "/usr/bin/nohup /home/jkl/.odrive-agent/bin/odriveagent > /dev/null 2>&1 &"

Start the background service

startodrive

Goto odrive site -> settings -> get your token

odrive authenticate your_auth_token

Link your local folder with corresponding remote drive/folder

 odrive mount /media/jkl/64BCC6E1BCC6ACBC/Users/jkl/Google\ Drive/ /jagadkanihal

 

backup (send from local to remote)

odrive backup /media/jkl/64BCC6E1BCC6ACBC/Users/jkl/Google\ Drive/ /jagadkanihal

 

sync (remote to local)

Extension meanings

  • .cloudf  : cloud folder
  • .cloud : cloud file
odrive sync jagadkanihal.cloudf

 

3. Duplicati (compressed/encrypted one way backup : local to remote)

add backup -> step2. destination -> click on the authid (opens drive – login – fetches auth id) -> path on server should folder name (if left blank, it will choose the main/login folder )

Duplicati

clipboard Append

Appending content to clipboard without overwriting
Here is the script which essentially does this.

clip_append_n.sh

#!/bin/bash
#text under selction i.e. highlighted
p=$(xclip -selection primary -o)

# text already in the clipboard , (what happens if there is image in clipboard)
c=$(xclip -selection clipboard -o)

#seperator
sep=$'\n'
echo "$c$sep$p" | xclip -selection clip
notify-send --expire-time=1000 "ClipAppend" "Appended with newline"

#ubuntu ignore that expire time option hence - workaround hack
sleep 1
pkill notify-osd

Add keyshortcut to run this script:
System settings>> Keyboard and click on Shortcuts tab. Click on Custom Shortcuts and then click on + sign to define your key combination. In the command box put:

clip

bash -c 'bash ~/clip_append_n.sh '

click on disabled on the right side of the entry , you’ll get new accelerator visible then put your desired “KEYS” for the shortcut

I use ALT+N to trigger the action appending highlighted content to clipboard with a newline as a seperator.

References:

http://askubuntu.com/questions/797132/how-to-only-append-to-clipboard-but-not-overwrite-current-data

http://askubuntu.com/questions/873948/automatically-modify-clipboard-content-regex-pattern-replacement-when-copy-pas?noredirect=1&lq=1

http://stackoverflow.com/questions/42507916/how-can-access-read-and-write-system-clipboard-from-a-kernel-module

Linux Filesystem Hierarchy Standard (FHS)

man hier :[hierarchy] gives all this info

/boot   ->  boot related files, linux kernel binary, grub config etc

/etc     -> [backronym “Editable Text Configuration”] [sysad related] all host specific system wide configuration files for applications , eg network manager , lightdm everything is here &  startup, shutdown, start, stop shell scripts for every individual programs.

    • /etc/bash.bashrc : Contains system defaults and aliases used by bash shell.
    • /etc/profile : Bash shell defaults
    • /etc/profile.d : Application script, executed after login.
  1. /etc/skel : Script that populates new user home directory. (by mistake you deleted  local configuations in ~/*.*)
  2. /etc/crontab : A shell script to run specified commands on a predefined time Interval.
  3. /etc/init.d : Service startup Scripts.
  4. /etc/resolv.conf : Domain Name Servers (DNS) being used by System.
  5. /etc/network/interfaces for static/dhcp ip address, LAN configuration
    • /etc/hosts : Information of Ip addresses and corresponding host names.
    • /etc/hosts.allow : List of hosts allowed to access services on the local machine.
    • /etc/host.deny : List of hosts denied to access services on the local machine
  6. /etc/grub.conf : grub bootloader configuration file.(/boot/grub/grub.cfg).
  7. /etc/modules: Configuration files for system modules.
  8. /etc/passwd : Contains password of system users in a shadow file, a security implementation.
  9. /etc/X11 : Configuration files of X-window System.
  10. /etc/fstab : Information of Disk Drive and their mount point.  (command – blkid )
  11. /etc/mtab : Currently mounted blocks information. (??usecase)
  12. /etc/group : Information of Security Group.(usecase??)

/proc -> [Process Information] This is a pseudo filesystem(procfs) contains information about system processes/info,kernel/module exported variable values, for running process  /proc/{pid}

  1. /proc/cpuinfo : CPU Information
  2. /proc/filesystems : File-system Information being used currently.
  3. /proc/interrupts : Information about the current interrupts being utilised currently.
  4. /proc/ioports : Contains all the Input/Output addresses used by devices on the server.
  5. /proc/meminfo : Memory Usages Information.
  6. /proc/modules : Currently using kernel module.
  7. /proc/mount : Mounted File-system Information.
  8. /proc/stat : Detailed Statistics of the current System.
  9. /proc/swaps : Swap File Information.
  10. /proc/kallsyms : all kernel symbols that have been loaded
  11. /proc/version : os , kernel version info

/var –>  [Variable Files] Content of the files that are expected to grow can be found under this directory.

  1. system log files (/var/log/*)[sysad related]
  2. packages and database files (/var/lib);
  3. emails (/var/mail);
  4. print queues (/var/spool);
  5. lock files (/var/lock);
  6. temp files needed across reboots (/var/tmp);
  7. webserver files (/var/www)
  8. cache file (/var/cache)

/dev –> [Device Files] Essential device files that are needed for the system.

special file that appears in filesystem. It is an interface for the device driver.They allow software to interact with a device driver using standard input/output system calls, which simplifies many tasks and unifies user-space I/O mechanisms.

2 types of devices : char devices and block devices

terminal devices, usb etc. for eg: /dev/tty1, /dev/usbmon0

  1. stdin -> /proc/self/fd/0
  2. stdout -> /proc/self/fd/1
  3. stderr -> /proc/self/fd/2

psuedo devices

  1. /dev/null – accepts and discards all input; produces no output (always returns an end-of-file indication on a read) (use this as a sink)
  2. /dev/zero – accepts and discards all input; produces a continuous stream of NUL (zero value) bytes (use this as source)
  3. /dev/full – produces a continuous stream of NUL (zero value) bytes when read, and returns a “disk full” message when written to
  4. /dev/random and /dev/urandom – they produce a variable-length stream of pseudo-random numbers.

/sbin       ->  [system binaries] essential os system binaries      e.g., fsck, init, route, insmod, rmmod, modinfo, modprobe

/bin          ->  [user binaries] command line binaries for single user mode?? like cat, cp, mkdir, su (permissions rwx, s-> secure execute for root user)

/lib*      ->  [system libraries] for /bin/ & /sbin/  binaries(system wide share/usr/bin?)

/usr    ->   Secondary hierarchy for read-only user data; contains the majority of (multi-)user utilities(libraries, docs) and applications[apt-get installs packages to /usr/bin and /usr/lib]

  • /sbin -> non essential system binaries for sysads.For example: cron, sshd, useradd,
  • /bin -> non essential command binaries for all users.For example: at, awk, cc, less, scp
  • /lib* ->  Libraries for the binaries in /usr/bin/ and /usr/sbin/.(above)
  • /include -> Standard include files for c,c++(gcc) programs etc.
  • /src -> kernel source code with header files
  • /share ->Architecture-independent (shared) data. man files, info files etc
  • /games -> some games
  • /local ->Tertiary hierarchy for local data, specific to this host.users programs that you install from source.Typically has further subdirectories, e.g., bin/, lib/, share/
    • /bin ->(compiled binaries) openssl, etc
    • /lib -> (shared libraries for binaries in /usr/local/bin)libssl.so , libopencv
    • /include -> header files for the apps/pkg installed in /usr/local
    • /etc -> configuration file for apps in /usr/local/bin
    • /share -> man, info pages for apps/pkgs in /usr/local/

/media    ->  [Removable media] Temporary mount directory for removable devices.eg:/media/cdrom

/mnt         ->  [mount point] Temperoray mount directory, sysads can mount different filesystems  here.


/sys  ->  [system information] Sysfs is mounted under the /sys mount point.The sys filesystem is the location where information about devices, drivers, and some kernel features is exposed.

/tmp  ->  [temporary files] deleted on reboot. where as /var/tmp are preserved during reboot.

/opt -> [optinal ] Optional independent (stand alone) application software packages. (just like in windows c:/programfiles/) eg:/opt/google/

/run -> [Runtime info] Information about the running system since last boot, e.g., currently logged-in users and running daemons.

/srv-> [service] Site-specific data served by this system, such as data and scripts for web servers, data offered by FTP servers, and repositories for version control systems.usecase??


/root    ->  [root:~] root user’s home directory

/home -> [jaggi:~] Home directories for all users to store their personal files and personal settings eg: /home/jaggi


References – (pics, some text is directly copied from these sites)

https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

https://en.wikipedia.org/wiki/Device_file

http://www.tecmint.com/linux-directory-structure-and-important-files-paths-explained/

http://www.thegeekstuff.com/2010/09/linux-file-system-structure

linux-directory-structure

blprb

Set up qtcreator for kernel module developement (kernel programming)

linuxpenguinqt

Why use qtcreator?
qtcreator is one of the best ide for c/c++ developement. It has amazing autocompletion features. It makes code walktrough of large projects (like linux kernel code) easier.

1.Import existing makefile project into Qt Creator
If you are using your own build system, you create a project by selecting “Import of Makefile-based project”. This will create some files in your project directory including a file named <project_name>.includes  . In that file, simply list the paths you want to include, one per line. Adding “include paths” to qtcreator for kernel projects  enables autocompletiton for kernel related sources (module.h ) on Qt creator.:

All this really does is – tell Qt Creator where to look for files to index for auto completion. Your own build system will have to handle the include paths in its own way.

for eg. for project ‘dummy’ , the include file would be ‘dummy.includes’ which should contain path to kernel source

/usr/src/linux-headers-4.4.0-92-generic/include/

Reference:
http://stackoverflow.com/questions/2752352/how-to-add-include-path-in-qt-creator

 

2. Makefile and build(compile)

Example Makefile (for module file jaggi-klm.c file)

obj-m += jaggi-klm.o 

all: 

        make -C /lib/modules/$(shell uname -r)/build M=$(PWD)/dumkp modules

clean: 

          make -C /lib/modules/$(shell uname -r)/build M=$(PWD)/dumkp clean

builds module file with extension .ko (kernel object)

 

3. Then do load/unload the module using:

sudo insmod  _____.ko
sudo rmmod   _____.ko

 

 

References:

iitb email on gmail/thunderbird

On Gmail App

CSE mail

https://www.cse.iitb.ac.in/~swlab/Thunderbird_CSE.pdf

There is a typo in smtp part, instead of imap it should be smtp

GPO mail

http://www.cc.iitb.ac.in/engfaqgeneinf/10-english/articles/28-information-on-setting-up-imap-based-email-clients

outgoing smtp starttls(all certificates accepted)

__________________________________________________________________________

On thunderbird

CSE mail (same as above)

https://www.cse.iitb.ac.in/~swlab/Thunderbird_CSE.pdf

There is a typo in smtp part, instead of imap it should be smtp

 

mail not sending? see

https://bugzilla.redhat.com/show_bug.cgi?id=1489966#c3

https://bugzilla.mozilla.org/show_bug.cgi?id=1371309

for setting logging  – see Linux/unix section of  https://wiki.mozilla.org/MailNews:Logging#Logging_prior_to_Thunderbird_55.0a1_.2F_SeaMonkey_2.52a1

GPO mail

http://www.cc.iitb.ac.in/engfaqgeneinf/10-english/articles/28-information-on-setting-up-imap-based-email-clients

incoming imap – ssl/tls port 143- plain password

 

____________________________________________________________________________

— gmail on thunderbird

https://productforums.google.com/forum/#!topic/gmail/3jO9r667zRg

https://askubuntu.com/questions/910614/why-cant-i-progress-beyond-entering-my-username-when-configuring-thunderbird-to/910630#910630

most probable cause iitb proxy/firewall not allowing mail ports to google

How to get most out of ML course (CS-725) @ iitb?

This blog post is meant for students at IIT Bombay considering to take Foundations of Machine Learning (CS-725) Course taught by prof. Ganesh Ramakrishnan.

Prerequisite: To do well in this course, you should be thorough with Linear Algebra (LA) and Probability Theory(PT) concepts.

If you are already familiar with LA and PT, you can skip this part.

For Linear Algebra (LA), I suggest you go through “Linear Algebra and Its Applications” Book by Gilbert Strang along with his MIT OCW videos  , Once you do this, you should be familiar with things like rank, row space, column space, null space of a matrix, least squares method(*), eigen values, and various decompositions of matrices like Eigendecomposition, SVD, QR, LU etc.

images  images-1

For Probability theory, go through “First Course in Probability” book by Sheldon Ross along with MIT OCW videos of prof. Tsitsiklis. Once you go through this, You should be thorough with Bayes rule(prior, posterior, likelihood), distributions like Gaussian(* very imp), binomial, beta etc

415tc8ue1l-_sx398_bo1204203200_  hqdefault-1

Extra prerequisite for non cs students, As this course involves 2 programming assignments and 1 project, you are expected that you should be able to use atleast one of  python/R/matlab languages effectively.

Now Coming to actual course content,

For 2016b(Autumn), the course content was properly organised compared to the previous offering(2016a Spring) of the same course. Before midsem, supervised regression models were covered  and post midsem supervised classification models and a little bit of unsupervised clustering models were covered.

2016a(Spring) course content is available here

2016b(Autumn) course content is available here

The main reading material for the course are class slides and tutorials that are given in the class. There are short lecture videos of class lectures that are available on course site on bodhi tree. Even if you miss the lecture, I would suggest you not to miss tutorial solutions discussion.

Whatever I mention below are for understanding and auxillary.

I personally preferred using “Pattern Recognition and Machine Learning” Book by Christopher Bishop as reference book. Along with the book, I used AI course videos of patrick H winston available @ mit ocw. These videos were particularly helpful for support vector machines(SVM), decision trees and neural networks. He is an amazing orator, conveys the content in simple terms without losing out on mathematical rigour. If you get time, I would suggest you to watch all his videos.

images-2

Before midsem:

For Linear, Ridge, Lasso, I studied mainly from online resources.

These are some of the material on youtube that I found useful (in no particular order).

mathmonk

Ritvik

patrickJMT

Alexander Ihler

Alex Smola

For support vector regression(SVR) – I think winston’s lecture video on svm + it’s mega recitation video would be sufficient

For KKT optimization conditions, This link  would be useful.

mercer kernels (I still haven’t completely understood these things, I’ll update once I find any good resource)

Post Midsem:

For perceptron, sigmoidal classifier (aka logistic regression):  class slides and tutorials should be sufficient

Here is a simple proof of convergence of perceptron that I found online.

For Neural Networks(*), CNN, RNN, LSTM: I started with  Patrick H winston’s AI video on neural which gives the background and overall idea of a neural network and back propagation which is a must for understanding the working of neural networks.

Once you go through the Patrick’s video, you can watch Andrej Karpathy’s CS231(stanford) videos, this guy and his blog is amazing . I actually binge watched all videos in that playlist over 2 days, really enjoyed it.

For Decision Trees – Adaboost, Support Vector Classification : Again Patrick’s videos would suffice along with slides.

For unsupervised learning: (again haven’t quite understood this one also, will update this one once I find any good resource for this)

I just found this Presentation by Yann LeCun contains overview of lot many topics ML, DeepLearning (differences + overlap -> slide 24) http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf 

LAMP and XDEBUG setup on ubuntu

Use tasksel to install LAMP

16.04 – https://askubuntu.com/a/914181/650264

sudo apt install tasksel
tasksel 

and xdebug

sudo apt install php-xdebug

xdebug configuration on the server

cd /etc/php/7.0/fpm/conf.d/
#/etc/php/7.0/mods-available/xdebug.ini
sudo vim 20-xdebug.ini
zend_extension=/usr/lib/php/20151012/xdebug.so
xdebug.idekey=PHPSTORM
xdebug.remote_enable = 1

;one of these should be set
;set this where ide is running(check ifconfig and put below) and for any client - android etc
;xdebug.remote_host = 192.168.0.101 (won't work, should be a unique ip in the network of server,wifi temp ips wont work)
xdebug.remote_host = 10.1.224.101

;set this for ide running anywhere (roaming laptop) and client calling must also be the machine\
;(because xdebug uses request to determine the ip of ide to send debug info)
;xdebug.remote_connect_back = 1

xdebug.remote_port = 9000

;if set, debugs without that cookie requirement
xdebug.remote_autostart=1

xdebug.max_nesting_level = 400
xdebug.scream = 0
xdebug.cli_color = 1
xdebug.show_local_vars = 1

sudo service php7.0-fpm restart 
sudo service apache2 restart

xdebug setup for phpstorm ide as debug client

 

 

+Another Video