Gih's Blog

只言片语

A better way to backup redis' rdb snapshot.

2014-07-20 by gihnius, tagged as redis

I first post a relate article here: http://www.qufor.com/topics/529999c87cc1f87c30000004

Here is an update.

Basically, you can directly copy the file to a safe place, here is the redis document about persistence: http://redis.io/topics/persistence

Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete.

You can get the persistence info by running this command: 

echo 'info Persistence' | redis-cli 

You will get an output like the following

# Persistence
loading:0
rdb_changes_since_last_save:6
rdb_bgsave_in_progress:0
rdb_last_save_time:1406022719
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

See these lines: rdb_bgsave_in_progress:0 and rdb_last_bgsave_status:ok. Why I need to verify the bgsave status?

By default redis save the dataset every N seconds if there are at least M changes in the dataset, this can be changed from configuration:

save 60 1000

if redis data is changed very frequent, I'd like to get the latest snapshot, so I'd run bgsave before copying rdb file by this command:

echo bgsave | redis-cli

So here is the finally script to do the backup:

#!/bin/bash

## simple redis rdb backup script
## usage
## rdb-backup.sh rdb.path backup.dir bgsave.wait.seconds

rdb=${1:-"/var/db/redis/dump.rdb"}

backup_to=${2:-"/data/backup/redis/"}

wait=${3:-10} ## default wait for 10 seconds

test -f "$rdb" || {
    echo No rdb file found ; exit 1
}
test -d "$backup_to" || {
    echo Creating backup directory $backup_to && mkdir -p "$backup_to"
}

## launch bgsave
echo bgsave | redis-cli
echo "waiting for $wait seconds..."
sleep $wait
try=5
while [ $try -gt 0 ] ; do
    saved=$(echo 'info Persistence' | redis-cli | awk '/rdb_bgsave_in_progress:0/{print "saved"}')
    ok=$(echo 'info Persistence' | redis-cli | awk '/rdb_last_bgsave_status:ok/{print "ok"}')
    if [[ "$saved" = "saved" ]] && [[ "$ok" = "ok" ]] ; then
        cp "$rdb" "$backup_to"
        if [ $? = 0 ] ; then
            echo "redis rdb $rdb copied to $backup_to ."
            exit 0
        else 
            echo ">> Failed to copy $rdb to $backup_to !"
        fi
    fi
    try=$((try - 1))
    echo "redis maybe busy, waiting and retry in 5s..."
    sleep 5
done
exit 1

Available on github: https://github.com/gihnius/rdb-backup

Counting online users with Redis and Go

2014-07-20 by gihnius, tagged as go

an alternative way if you do not need to use websocket to do realtime counting.

only counting last 5 minutes, here assuming a user will stay online for five minutes after login/landing

var last_n_minutes = 5

// easily to change to yours
func redis_key_prefix() string {
    return App.Name + ":" + App.Env + ":"
}

func redis_key(suffix string) string {
    return redis_key_prefix() + suffix
}

// online key for online users every minute
// online_users_minute_5
// online_users_minute_6 // 5th and 6th minutes online users
func online_key(minute_suffix string) string {
    key := "online_users_minute_" + minute_suffix
    return redis_key(key)
}

// the key for THIS minute
func current_key() string {
    key := strconv.Itoa(time.Now().Minute())
    return online_key(key)
}

// return keys of last n minutes online users
func keys_in_last_n_minutes(n int) []string {
    now := time.Now()
    var res []string
    for i := 0; i < n ; i++ {
        ago := now.Add(-time.Duration(i) * time.Minute).Minute()
        res = append(res, online_key(strconv.Itoa(ago)))
    }
    return res
}

// add a online user to the set.
// call this operation from a ajax long pull is recommended, so
// do not need to write to redis every time user click/open a page.
func add_online_username(name string) {
    new_key := false
    key := current_key()
    if ok, _ := Redis.Exists(key); ok == false {
        new_key = true
    }
    Redis.Sadd(key, []byte(name))
    if new_key {
        // assuming a user will be offline after last_n_minutes
        expiredin := int64((last_n_minutes+1)*60)
        Redis.Expire(key, expiredin)
    }
}

// the online usernames
func online_usernames() []string {
    keys := keys_in_last_n_minutes(last_n_minutes)
    users, err := Redis.Sunion(keys...)
    if err != nil {
        return nil
    }
    var res []string
    for _, u := range users {
        res = append(res, string(u))
    }
    return res
}

// counting how many online users
// just do it from redis
func online_users_count() int {
    current_online_key := redis_key("online_users_current")
    keys := keys_in_last_n_minutes(last_n_minutes)
    Redis.Sunionstore(current_online_key, keys...)
    n, err := Redis.Scard(current_online_key)
    if err != nil {
        return -1
    }
    // go set_online_max(n)
    set_online_max(n)
    return n
}

// the max value of online users
func set_online_max(curr int) {
    max_online_key := redis_key("online_users_max")
    orig, _ := Redis.Get(max_online_key)
    n, _ := strconv.Atoi(string(orig))
    if curr > n {
        Redis.Set(max_online_key, []byte(strconv.Itoa(curr)))
    }
}

func get_online_max() int {
    max_online_key := redis_key("online_users_max")
    orig, _ := Redis.Get(max_online_key)
    n, err := strconv.Atoi(string(orig))
    if err != nil {
        return -1
    }
    return n
}

The redis package using here is: github.com/hoisie/redis, to install it to GOPATH:

go get -u github.com/hoisie/redis

Get the code and a full example, please check out: https://github.com/gihnius/redis_online_counter

A nice ORM library in common lisp

2014-07-20 by gihnius, tagged as lisp

Crane project home page: http://eudoxia0.github.io/crane/

(filter 'user) ;; Returns everything

(filter 'user :name "Eudoxia")(filter 'user (:> :age 21))

;; Returns a single object
(single 'user :name "Eudoxia");;Throws an error ifthis returns more
;; than one object(single!'user (:< age 35))

;; t if a match exists, nil otherwise
(exists 'user :name "Eudoxia");;Ifthis record doesn't exist create it
(get-or-create 'user :name "Eudoxia":age 19)

Ruby on Rails 人门前必看

2014-03-30 by gihnius, tagged as ruby
传送门:

https://www.codefellows.org/blogs/this-is-why-learning-rails-is-hard

rails

踩到 Bootstrap3 popover 的一个坑

2014-03-30 by gihnius, tagged as web

在实现显示投票用户的功能的时候(看这里(过期)) , 想把用户名直接用 bootstrap3 的 popover 展示. 需要在 popover 里面动态加载用户名列表.

开始时这么干的: 

html:

<a href="javascript:void(0);" class="voted-users-popover" ....>显示该选项用户</a>
<a href="javascript:void(0);" class="voted-users-popover" ....>显示该选项用户</a>
<a href="javascript:void(0);" class="voted-users-popover" ....>显示该选项用户</a>

js:

// setup popover
$("a.voted-users-popover").each(function() {
    var el = $(this);
    el.popover({
       html: true,
       content: function() {
           // call ajax and return the content html.
           // 这里 ajax call 应该是设置 async = false.
       },
    });
});

Read more »

如何优化这段代码

2013-04-15 by gihnius, tagged as ruby

code:

def insert(hash)
  cols = ""
  vals = ""
  hash.each do |c,v|
    unless v.nil?
      cols << "#{c},"
      if v.is_a?(String)
        vals << "'#{PGconn.escape(v)}',"
      else
        vals << "#{v},"
      end
    end
  end
  vals.gsub!(/,$/, '')
  cols.gsub!(/,$/, '')
  sql = "INSERT INTO twitter_users(#{cols}) VALUES(#{vals});"
  #puts_and_return sql
  begin
    db.async_exec sql
  rescue PG::Error => e
    puts "Oops... >>> in #{hash} : #{e}"
  end
end

table 已确保可以容纳 hash

这种情况在ruby2 我会用 Struct.

这个方法有什么不妥之处,或者有没有更好的方法?


cl-common-blog博客功能基本完善

2011-09-14 by gihnius, tagged as lisp

划了个 0.2 版本, 个人测试, 已经相当稳定了, 而且比现在的博客方便, 再过不久我就完全使用cl-common-blog. 博客地址改为 http://blog.gihnius.net/.
写一个博客太简单了, 很多lisp的特性/功能都没有用到, 可能还不如写一个emacs插件折腾人. 当然, 这么说也是基于自己对博客应用的定义, 如果想的复杂, 就可以做得非常复杂, 可是没用. 接下来, 我计划使用common lisp开发一些应用. Planing...

用guile重写emacs?

2011-09-07 by gihnius, tagged as emacs

在solidot 看到这个新闻, 关于前段的:GNU黑客大会2011 , 里面提到用 Guile 实现 Emacs .

Guile ? Scheme ? 写个 emacs ?我记得之前有新闻说 emacs 25 这个版本要用 common lisp 重写. 后来也发现一些 common lisp 做的非完整版本的 emacs 实现, 例如: climacs Hemlock 另外一些描述可以从这里看到:http://www.cliki.net/CL-Emacs现在怎么弄个 scheme 版本的? 我不是怀疑 scheme 不能写一个emacs, 我是觉得, 那些聪明人写了emacs之后, 会把scheme弄大吗? 会再来一个 浏览器, 播放器, 还有什么商业应用之类吗? 我希望他们会做, 然后再弄一个庞大的标准库, 然后: A Common Scheme Was Born!.Lisp社区有很多不稳定的因素, 严重分化阻碍了社区的健康发展. 看看 Paul Graham, common lisp玩得好好的, 却去弄个 Arc .

腾出桌面空间,把系统监视器之类的程序停掉.

2011-09-05 by gihnius, tagged as linux

像很多人一样,平时喜欢把一个系统监视器显示在桌面, 我使用的是 conky. 

桌面除了它之外, 就剩下编辑器,浏览器之类的主要程序了.
现在我把conky 卸载了. 我觉得桌面上太多的东西容易分散注意力. 现在别说 conky, 连任务栏都省了. 

写了一个简单的监控脚本, 通过espeak有问题时再语音通知我. 避免一直显示没用途的信息.espeak 是一个轻量级的 text-to-speech 引擎. 它使用特别的 语音波 保存声音, 通过 波-声音的转换发声, 而不是打开一个个大大的音频文件播放. 

因此, 使用它做一个系统通知绰绰有余, 不用占用太多的系统资源.首先把 notify-send 替换掉, 取而代之的这个脚本: (notify-say)

#!/bin/sh

## pass by env ESPEAK_ARGS or define here
espeak_args=${ESPEAK_ARGS:-"-a 150 -g 0.3 -p 55 -s 160 -b 1"}

## read from a fifo pipe
ttsf="/tmp/.${USER}.tts.fifo"

ps -aux | grep -q "[Tt]ail -f $ttsf" || (test -p $ttsf || mkfifo $ttsf)
ps -aux | grep -q '[Ee]speak' || (nohup tail -f $ttsf | espeak $espeak_args >/dev/null &)

[ $# -eq 1 ] && [ "$1" = '-stop' ] && {
     ps -aux | grep "[Tt]ail -f $ttsf" | awk '{print $2}' | xargs kill
     #killall espeak
     rm -f $ttsf
     exit
}

echo $* > $ttsf 
这样,系统中一直保留一个 espeak 实例, 通过向一个 FIFO PIPE 发送信息, espeak 会顺序朗读出来.在监控脚本中调用:
$ notify-say "Network connection lost"
## 改变参数 朗读中文

$ESPEAK_ARGS="${ESPEAK_ARGS} -v zh" notify-say "请注意, 主文件系统满了." 
其实, 系统通知并不多, 不用一直运行那些占资源的监视器!另外: 截一张conky的图, 做个留念吧!

用Common Lisp写的博客程序可用了!

2011-09-03 by gihnius, tagged as lisp

发布了源代码, 并部署到 dev.gihnius.net .

功能很普通, 只能算一个可运行的程序. 毕竟是第一次试手 lisp 的 web 应用.