Gih's Blog

只言片语

这么玩 lisp,实在让人蛋疼!

2014-09-08 by gihnius, tagged as lisp

取个环境变量都要这么折腾:

(defun getenv (var)
  #+cmu (cdr (assoc (if (symbolp var)
                        var
                        (intern var :keyword))
                    ext:*environment-list*))
  #+sbcl (sb-ext:posix-getenv (string var))
  #+lispworks (hcl:getenv var)
  #+clisp (ext:getenv (string var))
  #+allegro (sys:getenv (string var))
  #-(or cmu sbcl lispworks clisp allegro)
  (error "GETENV not implemented for your Lisp platform."))


Speeding up fsck on the disk of TimeMachine

2014-09-06 by gihnius, tagged as mac

Why fsck?

Sometimes the file system get corrupted and hence you may unable to boot your system. In such case you will need to repair the corrupted file system. In Linux you can repair it using fsck command. In windows we have command chkdsk. In OSX we have "fsck", "fsck_hfs".

There are several reasons behind the file system corruption. For example, improper shutdown, suddenly cut off the power supply, a storage device was removed when system is in process to write the data on it, accidental system file deletion and viruses can cause file corruption and unstable system. Some file corruptions are less harmful and user can continue working. However it is important to run a “fsck” once in a while as a practice.

Why time machine?

Time Machine is Apple's built-in backup solution for OS X that creates hourly backups of all files on the system. It takes many snapshots of your local drive, copies many millions of files to the backup disk volume.  By default, Checking the time machine disk volumes with Disk Utility,  can be painfully slow, taking many hours or days to complete, if it completes at all.

How?

Find the time machine disk, open Terminal.app, type diskutils list, find the line contains Apple_HFS YOUR_DISK_NAME, get the device name from the last column below IDENTIFIER, for example, disk3, disk4...

Run fsck_hfs with 2G memory cache:(adjust less than the system memory)

sudo fsck_hfs -f -y -c 2g /dev/disk3

The key is to increase the size of the cache used by fsck_hfs. see more from manual:

-c size Specify the size of the cache used by fsck_hfs internally.  Bigger size
        can result in better performance but can result in deadlock when used
        with -l option.  Size can be specified as a decimal, octal, or hexadec-
        imal number.  If the number ends with a ``k'', ``m'', or ``g'', the
        number is multiplied by 1024 (1K), 1048576 (1M), or 1073741824 (1G),
        respectively.

Learn about using Disk Utility to verify or repair disks.

Another way to make memoization function in Go

2014-09-03 by gihnius, tagged as go

Cache any function calls in Go without using reflect, very simple idea, use a normal memory cache.

Here is what I do:

package memoize

import (
    "sync"
    "time"
)

type memo struct {
    Timeout time.Time
    Result interface{}
}

type MemoPool struct {
    Pool map[string]*memo
    mutex *sync.RWMutex
}

var mp *MemoPool

func init() {
    var m = map[string]*memo{}
    mp = &MemoPool{Pool: m, mutex: new(sync.RWMutex)}
}

// memorize result return from caller() block, timeout in N seconds
func Memoize(key string, caller func() interface{}, timeout uint) interface{} {
    if timeout == 0 {
        // do not memoize
        return caller()
    }
    mp.mutex.RLock()
    memoized := mp.Pool[key]
    mp.mutex.RUnlock()
    // reached timeout or not memoized
    if memoized == nil || memoized.Timeout.Before(time.Now()) {
        result := caller()
        if result != nil {
            duration := time.Duration(timeout) * time.Second
            mp.mutex.Lock()
            mp.Pool[key] = &memo{
                Timeout: time.Now().Add(duration),
                Result: result,
            }
            mp.mutex.Unlock()
        }
        return result
    }
    return memoized.Result
}

func UnMemoize(key string) {
    delete(mp.Pool, key)
}

func UnMemoizeAll() {
    for key, _ := range mp.Pool {
        delete(mp.Pool, key)
    }
}

Check out the code from github: https://github.com/gihnius/gomemoize

usage

import "memoize"
// cache a function call for TimeOut (>= 0) seconds
memoize.Memoize("Cache_Key", func() interface{} {
  cache_return_value, err := your_function(args)
  if err != nil {
    // handle err
    // won't cache if return nil
    return nil
  }
  return cache_return_value
}, TimeOut)

// save result
result := memoize.Memoize("Cache_Key", func() interface{} {
  // call your function(s)
  cache_return_value, err := your_function(args)
  if err != nil {
    // handle err
    // won't cache if return nil
    return nil
  }
  return cache_return_value
}, TimeOut).(your_function_return_type)

// cache a long time calc subroutine
func calc_sub(i int, s string) (string, err) {
  // ...
}
result := Memoize("calc_sub", func() interface{} {
  res, err := calc_sub(1000, "string to process")
  if err != nil {
    return nil // not memoized
  }
  return res
}, 60) // cache for 1 minutes

// re-fetch the memoized(cached) result with the same call.

// remove a cached call
memoize.UnMemoize(key_name)
// remove all cached calls
memoize.UnMemoizeAll()

test

git clone https://github.com/gihnius/gomemoize
cd gomemoize
./test.sh
# or
VERBOSE="-v" ./test.sh

It works well but not so functional.

More about  Memoization from Wikipedia.


Delete all but the most recent N files(objects) from s3.

2014-08-22 by gihnius, tagged as ruby

When I backup servers and save data to AWS S3, usually just upload entire files to S3. This is done automatic by the backup script. The uploading process looks simple:

S3_PUT backup_data_a s3://org-sys-backup/
S3_PUT backup_data_b s3://org-sys-backup/
S3_PUT backup_data_c s3://org-sys-backup/
S3_PUT backup_data_d s3://org-sys-backup/
S3_PUT backup_data_e s3://org-sys-backup/
S3_PUT backup_data_f s3://org-sys-backup/
...

## or 
S3_PUT backup_data_20140801xxxx s3://org-sys-backup/
S3_PUT backup_data_20140802xxxx s3://org-sys-backup/
S3_PUT backup_data_20140803xxxx s3://org-sys-backup/
S3_PUT backup_data_20140804xxxx s3://org-sys-backup/
S3_PUT backup_data_20140805xxxx s3://org-sys-backup/

I don't care the backups' data older than 1 week, then to free the space in S3 I have to delete those old data files (called objects in S3), just keep the most recent 5 copies. There maybe ready-made tools available,  but I want to roll my own. I wrote a ruby script to do that, here is source code:

#!/usr/bin/env ruby

## Delete all but the most recent N files from s3://bucket
## supply a prefix and count of copies and the bucket name

require 'right_aws'
require 'optparse'

opts = {}
OptionParser.new do |o|
  o.on("-a", "--access-key-id X", "aws ec2 access key id") do |x|
    opts[:aws_access_key_id] = x
  end
  o.on("-s", "--secret-access-key X", "aws ec2 secret access key") do |x|
    opts[:aws_secret_access_key] = x
  end
  o.on("-b", "--bucket X", "bucket name") do |x|
    opts[:bucket] = x || ""
  end
  o.on("-p", "--prefix X", "object name prefix") do |x|
    opts[:prefix] = x || ""
  end
  o.on("-c", "--count N", "how many copies to keep") do |x|
    opts[:count] = x || 1
  end
  o.parse!
end

if opts[:bucket].size < 1 || opts[:prefix].size < 1
  puts "bucket or prefix not valid!"
  exit 1
end

s3 = RightAws::S3.new(opts[:aws_access_key_id], opts[:aws_secret_access_key])

object_keys = {}

s3.interface.incrementally_list_bucket(opts[:bucket], {'prefix' => opts[:prefix], 'delimiter' => '/'}) do |item|
  item[:contents].each{|c| object_keys[c[:key]] = c[:last_modified] }
end

## sort obj keys by time(:last_modified)
sorted_keys = object_keys.sort_by{|k,v| Time.parse(v).to_i}.map{|x| x[0]}
puts "Found #{sorted_keys.size} objects with prefix: #{opts[:prefix]}"

if sorted_keys.size > opts[:count].to_i
  old_keys = sorted_keys.first(sorted_keys.size - opts[:count].to_i)
  n = old_keys.inject(0) do |a, k|
    s3.interface.delete(opts[:bucket], k)
    a += 1
  end
  puts "Deleted #{n} old objects."
end

This script get the objects list which object name match the given prefix, and sort them by modified time, then find the old ones and delete them.

Save it as, eg. s3_cleanup.rb.

How to run it?

You need to install a ruby gem 'right_aws' by: gem install right_aws, then

ruby s3_cleanup.rb -a 'access-key-id' -s 'secret-access-key' -b org-sys-backup -p 'backup_data_' -c 5

an example output:

Found 8 objects with prefix: backup_data_
Deleted 3 old objects.

This script is available in Github.

You may want to read about Object Expiration before checking out the script. That maybe a  official recommended practices.

Amazon S3 Announces Object Expiration Amazon S3 announced a new feature, Object Expiration that allows you to schedule the deletion of your objects after a pre-defined time period. Using Object Expiration to schedule periodic removal of objects eliminates the need for you to identify objects for deletion and submit delete requests to Amazon S3. You can define Object Expiration rules for a set of objects in your bucket. Each Object Expiration rule allows you to specify a prefix and an expiration period in days.The prefix field (e.g. "logs/") identifies the object(s) subject to the expiration rule, and the expiration period specifies the number of days from creation date (i.e. age) after which object(s) should be removed. Once the objects are past their expiration date, they will be queued for deletion. You will not be billed for storage for objects on or after their expiration date.

Here is another related: Managing Lifecycle Configuration.

This is my desktop... 5 years ago.

2014-08-15 by gihnius, tagged as freebsd, linux

Can you guess which OS?

Emacs, Firefox, Xterm, Rxvt, Fcitx...

and FreeBSD!

A custom power management settings

2014-08-14 by gihnius, tagged as mac

pmset is a command line tool that apple has bundled in their OS X operating system. Its purpose is to display and/or modify the power management settings. Here is a modified settings in order to: 1. Save battery power as much as possible; 2. let you return to work as soon as possible.

battery mode (pmset -b)

Write the hibernation image to disk and powering off memory for Standby after 6 hours.

pmset -b standbydelay 21600

Sleep after 10 minutes idle.

pmset -b sleep 10

Spindown disk after 30 minutes idle.

pmset -b disksleep 30 

Set hibernatemode to 3. The system will store a copy of memory to persistent storage (the disk), and will power memory during sleep. The system will wake from memory, unless a power loss forces it to restore from disk image.

pmset -b hibernatemode 3

Turn down display after 5 minutes idle.

pmset -b displaysleep 5

Poweroff after 8 hours to save power.

pmset -b autopoweroffdelay 28800

charger mode (pmset -c)

In charger mode, I would like to change hibernatemode to 0.  So that the system can be waked up quickly. In this mode, the system will not back memory up to persistent storage. The system must wake from the contents of memory, the system will lose context on power loss. 

pmset -c hibernatemode 0

Standby, the manual said, standby only works if hibernation is turned on to hibernatemode 3 or 25. Here I set it the same as battery mode

pmset -c standbydelay 21600

Sleep after 10 minutes idle.

pmset -c sleep 30

Turn down display after 10 minutes idle.

pmset -c displaysleep 10

Spindown disk after 60 minutes idle.

pmset -c disksleep 60

Poweroff after 12 hours to save power.

pmset -c autopoweroffdelay 43200

Show custom settings for all power sources (pmset -g)

pmset -g custom

Put together and save to a script.

#!/bin/sh

## battery mode
pmset -b standbydelay 21600
pmset -b sleep 10
pmset -b disksleep 30
pmset -b hibernatemode 3
pmset -b displaysleep 5
pmset -b autopoweroffdelay 28800
## charger mode
pmset -c standbydelay 21600
pmset -c sleep 30 
pmset -c hibernatemode 0
pmset -c displaysleep 10
pmset -c disksleep 60
pmset -c autopoweroffdelay 43200

pmset -g custom

for more detail about pmset, please check out the manual page of pmset by man pmset.

tested on osx 10.8 - 10.9.4 


Setup a DNS cache server using dnsmasq

2014-08-08 by gihnius, tagged as internet

Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It is designed to provide DNS and optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS. 

apt-get update
apt-get install dnsmasq

if installed successful, test it by:

dig google.com @localhost

Setup local caching

Edit /etc/dnsmasq.conf (default location in debian linux).

#listen-address=0.0.0.0 ## default 
port=53

no-poll

no-resolv

#forwarding google dns and opendns
server=8.8.8.8
server=8.8.4.4
server=208.67.222.222
server=208.67.220.220

cache-size=1024

neg-ttl=7200

max-ttl=86400

interface=eth0

interface=lo0

In order to configure dnsmasq to act as cache for the host on which it is running, put  nameserver 127.0.0.1 in /etc/resolv.conf to force local processes to send queries to dnsmasq. 

in /etc/resolv.conf:

nameserver 127.0.0.1

then restart dnsmasq. /etc/init.d/dnsmasq restart.


DNS (UDP) tunneling by SSH with socat.

2014-08-08 by gihnius, tagged as internet

Intro

In China, many "ISP" sucks. Their DNS servers often return incorrect ip address results,  is known as DNS poisoning! DNS poisoning is a common and simple way to stop people reaching correct web pages.

Here is a solution to get the correct DNS queries results.

Dependent tools

  • Server
    • A VPS server that can access famous public DNS servers correctly, eg. 8.8.8.8 (google dns) or 208.67.222.222 (opendns).
    • SSH server running on that VPS. (Please google: how to setup ssh server)
    • socat (Socket Cat). (Please google: how to setup or install socat)
    • dnsmasq (Optional, for caching).
  • Local
    • SSH client
    • socat (Socket Cat)
    • dnsmasq (Optional, for caching).

ssh, socat, dnsmasq are open source softwares which can be found and installed easily.

Samples and Steps

  • Server
    • Setup a DNS caching server using dnsmasq. (Optional)
    • If no local dns server, just use a public dns server instead, eg. 8.8.8.8:53
    • Forwarding UDP to TCP by socat (listen on port: 15353)
      • install socat
      • start socat:
        • if use a public dns server, eg. 8.8.8.8:53
          socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:8.8.8.8:53
        • if use local dns caching server: 127.0.0.1:5353
          socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:127.0.0.1:53
    • You can check the forwarding dns server using command line:
      dig +tcp google.com @127.0.0.1 -p 15353
  • Local
    • Setup SSH tunnel 
      ssh -N -L 15353:localhost:15353 username@vps.ip
    • Forwarding TCP to UDP by socat
      • if no local dns caching server, you can forward to port 53
        socat udp-recvfrom:53,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
      • of cause can forward to any port that can be used.
        socat udp-recvfrom:15353,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
    • Setup local dns caching server (Optional but recommend). See the server instruction above.

OK! 

Oh not yet!

ssh (tunnel) is not always working well! WTF!

Strip chars from a string in Go.

2014-07-22 by gihnius, tagged as go
import (
    "fmt"
    "strings"
)

func stripchars(str, chr string) string {
    return strings.Map(func(r rune) rune {
        if strings.IndexRune(chr, r) < 0 {
            return r
        }
        return -1
    }, str)
}

To clean up old packages in quicklisp

2014-07-22 by gihnius, tagged as lisp
(map nil 'ql-dist:clean (ql-dist:all-dists))