Wednesday, January 4, 2017

Package 'colr': Selecting Columns in 'R'


If you develop in 'R', like me, you may find yourself frequently selecting named data from  a data set, such as a data frame or matrix. 'R' supplies many ways to do this, but the syntax depends on the data type. So I wrote this package (named 'colr') with one syntax for all types. It uses a very powerful syntax called 'regex'. You may not be familiar with 'regex', but it is a DSL for text find and find and replace, implemented across a very wide range of languages, and 'R' has a full distribution of the 'Perl' 'regex' engine in the base package.

You can use this new package in a simple way with limited knowledge of 'regex', but if you learn more of 'regex' you can also use it in complex situations. The package is available on CRAN and in 'R' you can install it with install.packages('colr')
The easiest example is when you want to find all columns that have some specific string in several column names, such as all columns with 'length' in the column name. With this package that can be done with: cgrep(x, 'length') . I hope you give 'colr' a try and let me know what you think.

Functions provided in 'colr'

Package 'colr' provides two functions: 'cgrep' and 'csub'. 'cgrep' selects all columns where the string matches the column names, 'csub' changes the column names according to the replacement. The syntax for both function is identical except that csub takes one more argument then cgrep.

cgrep

'cgrep' is a function to select columns or rows from a dataframe, list or matrix with named columns. For lists the selection is by names of the list. cgrep will act only on the top level of the list. The return type is the same as the input type with the selected columns, or if the input was a list or matrix and only a single column was selected a flattened list or vector is returned.

csub

csub is a function to change column- or row names in a dataframe, list or matrix with named columns. The return is the same dataframe, list or matrix where the all column names with a match are changed to the pattern, or if no match was found the returned data frame, list or matrix is exactly as the input. For lists csub acts on the names in top level of the list.

regex

'Regex' is a DSL (grammar) for pattern matching and replacement in strings. A pattern consist of a quoted string of characters and meta characters. The quote in 'R' regex is ". The meta characters have special meaning in the grammar. An example of a meta character is the '.' (the dot) that stands for any character except a newline in the string. So to match in the string 'The quick brown fox jumps over the hedge' either 'ow' or 'ox' or 'ov' you can use the pattern "o.". There is much more to say about 'regex'; the vignette of the package 'colr' will give you some introduction. I find it most instructive to do 'regex' by example, I use the website 'regexr.com' for that.

Comparison

How is this package different from other methods. I frequently use the 'dplyr::select' function that works on data frames but not on other types. Let's compare a simple selection from the Iris dataset (a data frame). Use of the 'magrittr' style and of the relevant packages is implied in the examples.
with 'colr':
iris >%> cgrep("Sepal") >%> head ()
with 'dplyr': 
iris %>% select(contains("Sepal")) %>% head()
using subsetting:
iris %>% .[c('Sepal.Width', 'Sepal.Length')] %>% head()
As you can see the cgrep function from 'colr' is concise (it is also precise but I don't demonstrate that here).

Let's do an example with find and replace.

with 'colr':
iris >%> csub("Width", "Girth") >%> head()
with 'dplyr':
iris  %>% select(Sepal.Girth=Sepal.Width, Petal.Girth=Petal.Width)  %>% head() 
 using subsetting:
names(iris)[names(iris) %in% c("Petal.Width","Sepal.Width")] <- c("Petal.Girth","Sepal.Girth")
Again 'colr' is much more concise.

Other

Under the hood 'colr' uses 'grep' and 'gsub'. That will give the following code:
{x <- iris
names(x) <- gsub("Width", "Girth", gsub("Width", "Girth", names(x)))
x}

You could use that too, but it is much longer and it will only work on data frames. csub (and cgrep) will work just as fine on numeric types (such as matrices) and on lists.  I hope you give 'colr' a try and let me know what you think. 

Wednesday, July 13, 2016

Install and run Atom

It took me a while to get Atom (the editor) running on my chromebook, it turned out the solution was simple. The solution is actually not in the setup but in the startup of Atom from the command prompt in the chroot.

Installation

The installation progresses as usual. There is a source for installing on ubuntu trusty here: https://codeforgeek.com/2014/09/install-atom-editor-ubuntu-14-04/. Alternatively you can download de deb file from Atom.io and install with sudo dpkg -i atom.atom64.deb, or whatever name the downloaded package has from the command prompt after you have entered the chroot (from the crosh first shell and then enter-chroot).

You can know open atom from within your graphical interface, if installed, but it was more tricky to start it with xiwi. Atom, when startting from the command line, expects to be able to read the directory and apparantly has more than one mode. 


Run in a browser tab

So to start Atom in a browser window, go to the directory that you want to edit (that means you have the rights to do that as well) and then issue the command:

xiwi -T atom -f .

from you chroot. The -f option signals to atom to run in the foreground, the . (dot) tells atom to open the current directory for editing.

Wednesday, June 8, 2016

Restore a chroot to a different machine

On a chromebook you can set up chroots. This is possible through crouton. Chroots are separated environments that run all your linux software on the same kernel as the host machine. The result is you can take a chroot and copy it to a completely different machine and everything will run as before. That is just a fantastic and super easy way to keep all the environments on different machines in sync. So I routinely copy over chroots from one to the other machine, here is how to do that.

I use a chromebox and a chromebook. The chromebox is faster and has a better internet connection (over ehternet). So I tend to set up the chroot on the chromebox and then copy it to the chromebook. That is actually very easy.

The chromebox becomes the source machine and the chromebook is the target. The first step is to make a backup of your chroot on the source machine.

cd ~/Downloads
sudo edit-chroot -b trusty

Now copy the backup to a medium. I use a USB stick.

cp trusty-datestring-timestring.tar.gz  /media/removable/USB\ Drive/

That datestring-timestring will look like 20160607-2007 or similar.

On the target machine remove the old chroot if the name is the same, otherwise you can leave it there. Then insert the USB Drive and restore the chroot.

To remove the old chroot: sudo edit-chroot -d trusty

I assume you already have crouton on this device if not download it from https://goo.gl/fd3zc

cd ~/Downloads
sudo sh -e crouton -r trusty -f /media/removable/USB\ Drive/trusty-datestring-timestring.tar.gz 

That is all. You now have an identical chroot with all your software on the target machine.

I have set up virtual box on the source machine so when I fire up the chroot on the target I get this error.

/etc/rc.local failed with code 2

You can either set up the condition for virtual box (see my post from June 7) or get rid of the error. To get rid of the error make rc.local non executable with:

sudo chmod 644 /etc/rc.local

The error will not reappear. Obviously virtualbox will not be able to start on the target machine.

Written with StackEdit.

Tuesday, June 7, 2016

Setup Chromebook for development

June 7th, 2016.
This how I set up my chromebook, I do some development work, mainly front end and quantitative programming. I share this because some of it is hard to find out. This page owes a lot to a sources, I mention them in the text. All the work on crouton is borowed from David Schneider, but there other sources as well.

Go into developer mode

Depends on the machine on the Asus chromebox depress the reset button (next to the Kensignton lock) and the start the machine, nex release the reset button. After that pres Ctrl-D at the prompt (there is no hint on the screen). A good source for developer mode is: http://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices

Install the crouton extension

Crouton is an easy way to get access to a full linux setup on your chromebook, you will find everything you need here: https://github.com/dnschneid/crouton
First install the crouton extension, this is for integration with your browser. You can get it by this link: crouton extension.

Setup crouton

Download crouton (it is a script). You can get it here: https://goo.gl/fd3zc
Install crouton via these commands:
shell
sudo sh ~/Downloads/crouton -r trusty -t core,xiwi,keyboard,x11,cli-extra,extension,xfce
After it runs you can enter the chroot with sudo enter-chroot, this will get you into Ubuntu.

Setup software in Ubuntu

This section is borrowed from this website: https://medium.com/@martinmalinda/ultimate-guide-for-web-development-on-chromebook-part-1-crouton-2ec2e6bb2a2d#.mdjctop47 however the setup for nodejs is not functional if you want to use gulp. I have changed that section in the part below.
This part of the set up is done in the chroot, see above.
You may, of course choose, what you want to install, this is just what I figured out and wanted to share.

Git

sudo apt-get install git
Using version control is quite essential.

NPM + NodeJS

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs
Check your version with node -v, you can check you have the most recent on NodeJS.org.
For other nodejs versions, for example 0.12 use in stead:
curl -L http://deb.nodesource.com/setup_0.12 | sudo bash -
and then
sudo apt-get install nodejs
and then to compile and install native addons from npm you may also need to install build tools:
sudo apt-get install -y build-essential
To be able to install packages into global directory, we need to move it under home:
npm config set prefix '~/.npm-packages'
echo 'export PATH="$PATH:$HOME/.npm-packages/bin"' >> ~/.bashrc
//this should work now:
npm install -g bower grunt gulp ember-cli babel

Web starter kit

This is not really software the web starter kit from github on google/web-starter-kit is a template for web (apps). The dowload version of 0.6.3 does not seem to work well the cloned version does.
I found gulp with web starter kit (from google) does not run well with the latest gulp (3.9.1) so you may want to downgrade to 3.9.0. with
npm install -g gulp@3.9.0
This is a solution from Yeoman, see http://stackoverflow.com/questions/35832739/error-with-gulp-serve-requiring-external-module-babel-register/36475122#36475122

Sublime Text 3

Sublime is very popular, I am not a fan, I prefer Zed, a native app for chrome, you can get it here: https://chrome.google.com/webstore/detail/zed-code-editor/pfmjnmeipppmcebplngmhfkleiinphhp?utm_source=chrome-app-launcher
For sublime follow this section.
sudo apt-get install software-properties-common python-software-properties -y
sudo add-apt-repository ppa:webupd8team/sublime-text-3
sudo apt-get update
sudo apt-get install sublime-text-installer
At this it is reccomend installing Nautilus (a file browser) to get full support of gtk windowing system. Sublime fails to launch without it and on top of it, you get a GUI for file browsing. That maye be handy sometimes.
sudo apt-get install nautilus
Now you can launch Sublime with:
xiwi -t subl

Terminator

sudo apt-get install terminator
You can now launch terminator in a window with:
sudo xiwi -t terminator

Set up for virtual box

You may also want to run virtual box. I took these from the ubuntu/debian ppa for chromebook kernel headers. You can find it here: https://github.com/divx118/crouton-packages
Currently supported:
- Kernel 3.4.0 Architecture arm, i386 and am64 ==> These all need
testing.
- Kernel 3.8.11 Architecture arm, i386 and am64 ==> amd64 is
tested. i386 and arm needs testing.
- Kernel 3.10.18 Architecture arm
and amd64 ==> amd64 is tested. i386 and arm needs testing.
Open a crosh shell and follow on screen instructions:
cd ~/Downloads
wget https://raw.githubusercontent.com/divx118/crouton-packages/master/change-kernel-flags)
sudo sh ~/Downloads/change-kernel-flags
When running sudo sh ~/Downloads/change-kernel-flags -h it will give you the usage. You may want to revert the changes so put save a backup kernel use -r:
 sudo sh ~/Downloads/change-kernel-flags -r
Note: You will need to repeat the above steps after each chromeos update.
The following is done only once. Open a shell in your chroot:
cd ~
wget https://raw.githubusercontent.com/divx118/crouton-packages/master/setup-headers.sh

sudo sh setup-headers.sh
This script will: add a signing key with which the packages are signed, add the ppa to the sources.list.d install linux-headers-, install linux-image-, add a rc.local script to load vboxdrv modules on entering the chroot if they are available.
Kernel headers are build according to this guide https://github.com/dnschneid/crouton/wiki/Build-kernel-headers-and-install-Virtualbox-(x86) kernel headers for arm are cross compiled.
Now go ahead and install virtual box sudo apt-get install virtualbox

Install Rstudio

RStudio needs the latest r-base, not standard available in the ubuntu packages, so you need to add sources first. This method will use the latest development version of R-base from Michael Rutter.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 // wil add a key to your keyring
sudo add-apt-repository ppa:marutter/rdev

If you are met with the message: sudo: add-apt-repository: command not found. 
You can install it with 
sudo apt-get install software-properties-common
Then continue.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install r-base
After which you have r-base installed. You can now install R-studio if you like, download it from the Rstudio website and install using the debian package installer (in the GUI) if you have chosen the *.deb version.

Aliases

Aliases can save you a lot of time and energy. In your .bashrc add some lines, here is a few I use very frequently:
 alias cdd="cd ~/Downloads"
 alias x="xiwi -t"
 alias xw="xiwi -w"
There is also one handy alias that can allow you to get into chroot a bit faster. It has to be stored in .bashrc outside chroot:
echo 'alias ec="sudo enter-chroot"' >> ~/Downloads/.bashrc
And then leave chroot, and move it to the chronos home .bashrc:
logout
cat ~/Downloads/.bashrc >> ~/.bashrc
If you have just added these you need run source .bashrc to activate them.

Setup autostart on Chromebook

Move your crouton script and the backup into the Downloads directory on the machine (as well as all the keys and stuff). Specifically include crouton crouton.init crouton.conf and rw-rootfs.
After reboot run the file rw-rootfs (this is a script < see https://github.com/dnschneid/crouton/wiki/Autostart-crouton-chroot-at-ChromeOS-startup )
Run this file in the shell (not in the chroot)
like this: sudo sh ~/Downloads/rw-rootfs
It will reboot.
Next copy the crouton.conf file to /etc/init with this:
`sudo cp ~/Downloads/crouton.conf /etc/init/` 
Reboot and it will work. I use this to enter-chroot on startup

Backup and restore chroot

To backup your chroot use: sudo edit-chroot -b nameofchroot
To restore use sudo edit-chroot -r nameofchroot
You can delete a chroot by sudo edit-chroot -d evilchroot
In the event that you want to restore a backup to a stock chromebook (with developer mode enabled) you can use the crouton installer with the -f option specifying the backup’s tarball, e.g.,
sudo sh -e crouton -r raring -f /media/removable/SD\ Card/backup/raring-20130617-1234.tar.gz 

Update the software

When a new version of your chroot comes out you may want to upgrade.
Check for updates, download the latest version, and see what’s new by running croutonversion -u -d -c from the chroot (run croutonversion -h to see what those parameters actually do).
Exit the chroot and run sudo sh ~/Downloads/crouton -u -n chrootname. It will update all installed targets.

After an update of chomeos do

If you have virtual box installed, rerun the script to change the kernel flags. Open a crosh shell and follow on-screen instructions:
sudo sh ~/Downloads/change-kernel-flags
then reboot.
If you have set up autostart, rerun the file to make root writable this in the shell (not in the chroot)
like this: sudo sh ~/Downloads/rw-rootfs. It will reboot.
Next copy the crouton.conf file to /etc/init with this: sudo cp ~/Downloads/crouton.conf /etc/init/ if you have set up autostart.