Percent complete R function

R function to print percent complete.

If you use
method = message
you may want to use
suffix = ” % complete”

pctComplete <- function(
  progress = 0, last.progress = 0,
  total = 100, increment = 10,
  suffix = " ", method = cat, ...) 
{
  if (increment < 1)
    increment <- 1
  if (total < 0)
    total <- 100
  pct <- round(progress * 100 / total)
  if ( pct >= increment 
               + last.progress
               - last.progress%% increment )
  {
    method(paste("", pct, suffix, sep = ""),
           ...)
  }
  pct
}

 

aggregate free memory and swap

free -b reveals the total, used, free, shared, cache, and available memory.

Here’s the awk code to extract the total of the free physical memory and the free swap.

First, observe the free memory and swap, in bytes

# free -b | \
awk 'BEGIN{s=0}; NR<2{print $3}; NR>1{print $4}'
free
293203968
21441761280

Now, total the free memory and swap

# free -b | \
awk 'BEGIN{s=0};NR>1{n=$4;s=s+n;};END{print s}'
21741248512

The sum is slightly different because the second command was run at a different time.

Why I chose MDI during R installation

When installing R on a machine that is shared among several users, some of which are accustomed to the Mac OS, I had to choose between the Multiple Document Interface (MDI) and the Single Document Interface (SDI). The MDI article on Wikipedia pointed out that the MDI interface relates all documents to a common menu bar, which is what Mac users are accustomed to; Windows users probably have encountered both, since Office applications have used at times (depending on the generation) SDI and MDI. Therefore, to make the experience less alien for Mac users on a Windows machine, I installed R configured to use MDI.

Ubuntu temperature monitor

Background

I am trying to figure out how to monitor temperature on my Ubuntu system, and I found this thread:
https://ubuntuforums.org/showthread.php?t=2178610
So I started installing the packages recommended in response #5, but response #4 may be good, too.

Synopsis
$ sudo apt-get install lm-sensors hddtemp
$ sudo sensors-detect
$ sudo /etc/init.d/kmod start
[ ok ] Starting kmod (via systemctl): kmod.service.
$ sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +54.0°C  (crit = +120.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +53.0°C  (high = +100.0°C, crit = +100.0°C)
Core 0:         +52.0°C  (high = +100.0°C, crit = +100.0°C)
Core 1:         +51.0°C  (high = +100.0°C, crit = +100.0°C)

sed script for key:value to csv

I had files with one key-value pair per line, e.g.:

Acquired Name: foo
Acquired Date: 1066-10-14T10:00:00Z
etc.

I wanted to extract all of the fields to one CSV line per file.

This is the sed script that I used. As it happens, this would also work if all of the input files were concatenated (because I used the ‘d’ command rather than ‘q’).

/Acquired Name:/ { 
  s/.*Acquired Name: \(.*\)./"\1"/ ; h ; }
/Acquired Date:/ {
  s/.*Acquired Date: \(.*\)./"\1"/ ; x ; G ; s/\n/,/M ; h ; }
/Acquired Time:/ {
  s/.*Acquired Time: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/Job Code:/ {
  s/.*Job Code: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/Instrument:/ {
  s/.*Instrument: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/Bottle Number:/ {
  s/.*Bottle Number: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/Inlet Method:/ {
  s/.*Inlet Method: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/MS Method:/ {
  s/.*MS Method: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; h ; }
/Tune Method:/ {
  s/.*Tune Method: \(.*\)./"\1"/ ; x ; G ;
  s/\n/,/M ; p ; d ; }

The manual for sed
<https://www.gnu.org/software/sed/manual/sed.html>
explains the commands used:

  • d – Delete the pattern space; immediately start next cycle with next input line.
  • h – Copy the pattern space to the hold space
  • G – Append the hold space to the pattern space
  • p – Print the pattern space
  • s/pattern/replacement/ – Substitute pattern with replacement
  • s/pattern/replacement/M – Do multi-line substitution; see this
  • x – Exchange the pattern space and the hold space

 

colordiff

From http://linux.die.net/man/1/colordiff

colordiff is a wrapper for diff and produces the same output as diff but with coloured syntax highlighting at the command line to improve readability.

The output is similar to how a diff-generated patch might appear in Vim or Emacs with the appropriate syntax highlighting options enabled.

The colour schemes can be read from a central configuration file or from a local user ~/.colordiffrc file.

To enjoy on Ubuntu:

sudo apt install colordiff

Check these if you cannot ssh

In addition to checking that /etc/ssh/sshd_config is correct, you will want to check that the permissions on the user’s home directory and ~/.ssh are correct. I have gathered that:

  • The home directory must only be owned by AND only be writable by the login user.
  • The .ssh directory must only be accessible by the user.
  • The authorized_keys file must be only writable by the user.
[foo@foo-nas ~/.ssh]$ ls -la -D ""
total 14
drwx------  2 foo  foogrp    3  .
drwxr-x---  5 foo  foogrp   15  ..
-rw-r--r--  1 foo  foogrp  395  authorized_keys

ZFS expansion drive on Ubuntu

What ZFS has to offer for a laptop

ZFS is best used in a RAID configuration:

  • mirror (at least 2 disks)
  • raidz1 (at least 3 disks)
  • raidz2 (at least 4 disks)
  • raidz3 (5 or more disks)

A laptop does not typically offer the advantage of being able to house so many disks. Nevertheless, ZFS offers other advantages that make it sensible to use on a laptop:

  • scrubbing and on-line file system repair
  • snapshots
  • transparent compression
  • transparent encryption

I have two drives in my laptop:

  • a 256 GB SSD OS drive
  • a 512 GB 7200 RPM expansion drive

It might make sense at some point to replace the expansion drive with a higher capacity drive. For now, I installed ZFS onto the expansion drive, inspired by:
http://www.ibm.com/developerworks/library/l-zfs/index.html

Ideally, I would have a mirror between my OS drive and a partition on the expansion drive; however, not only is my expansion drive fairly limited, but also this is my first attempt at using ZFS beyond FreeNAS, which handles ZFS at arm’s length.  So, I decided to start with something simple: ZFS on an unmirrored disk.

Preliminary step

Although the kernel in Ubuntu 16.04 LTS supports ZFS, I needed to install the ZFS utils:
  apt-get install zfsutils-linux
to use the zpool and zfs commands.

What I should have done, perhaps

If I am reading the man page correctly for the zpool command (which says that the device name path is relative to /dev), I could have deduced the invariant id for my disk
  ata-HGST_HTS725050A7E630_RCF50ACE1P7B7M
using the command:
  ls -l /dev/disk/by-id
and then creating and mounting the “zpool” called “zpdrone” using the command

zpool create zpdrone disk/by-id/\
ata-HGST_HTS725050A7E630_RCF50ACE1P7B7M

What I actually did

Unfortunately for me, I hadn’t quite understood the man page, so I went with the first thing that worked:
  zpool create zpdrone sda

I next created a ZFS file system (“drone”) and enabled compression:
  zfs create zpdrone/drone
  zfs set compression=on zpdrone/drone

I proceeded to add some files to my file system (which was mounted at /zpdrone/drone), rebooted, and verified that I could see the files.

What went wrong, and how to fix it

Shortly thereafter, I booted with an external drive attached.
I found that the external drive was assigned to sda, so my expansion drive was assigned to sdb, and consequently my zpool didn’t get mounted; this gave me the opportunity to learn how to untangle the mess that I had caused.  As it turns out, this was not hard to do.

The first step was to “export” the pool (roughly equivalent to sync’ing and unmounting it):
  zpool export zpdrone

The second step was to “import” the pool using the disk ID and pool ID.  I obtained the pool ID (269414010796639802) from the output of
  zpool import
and then told zpool to import it by searching under
  /dev/disk/byid
for a zpool with the zpool ID

zpool import -d /dev/disk/by-id \
269414010796639802

Once again, my file system was mounted at
  /zpdrone/drone.

Subsequently, I could boot with an attached external hard drive, and, as expected,
  zpool status
showed the disk ID for the pool
(ata-HGST_HTS725050A7E630_RCF50ACE1P7B7M).

Programming R efficiently

I have begun a search for guidance on programming R efficiently.
I hope that I can revise this once I have posted it …

Here are the first URLs that I have found on the subject:


 

Off-topic but perhaps useful

Pyinotify – Linux filesystem watcher

from the Pyinotify wiki

Pyinotify is a Python module for monitoring filesystems changes. Pyinotify relies on a Linux Kernel feature (merged in kernel 2.6.13) called inotify. inotify is an event-driven notifier, its notifications are exported from kernel space to user space through three system calls. pyinotify binds these system calls and provides an implementation on top of them offering a generic and abstract way to manipulate those functionalities.

See the tutorial