TBlog 2017-06-07T03:00:23+00:00 yangyingchao@gmail.com Spark Ensime SBT Notes 2017-06-07T00:00:00+00:00 http://yangyingchao.github.io/Spark-Ensime-SBT-Notes

Table of Contents

简单记录一下折腾 Spark, Ensime, sbt 过程中遇到的问题。

sbt run 报错: Uncaught exception in thread SparkListenerBus

表现

15/02/06 15:56:49 ERROR Utils: Uncaught exception in thread SparkListenerBus
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:996)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:48)
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460)
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:46)
15/02/06 15:56:49 ERROR ContextCleaner: Error in cleaning thread
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:136)
    at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
    at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:134)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1460)
    at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:133)
    at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)

原因

SparkSession、SparkContext 没有关闭

解决方法

下面两者均可:

  • 添加 sc.stop() 或者 spark.stop()
  • build.sbt 里面添加:
    fork := true
    
]]>
Simple SVN Server On Gentoo 2017-04-10T00:00:00+00:00 http://yangyingchao.github.io/Simple-SVN-Server-On-Gentoo

Table of Contents

安装

emerge -av subversion

配置

emerge --config dev-vcs/subversion

之后进入 /var/svn/repos/conf 目录,编辑文件 svnserver.conf :

[general]
anon-access = none   #(匿名用户不可进)
auth-access = write  #(认证用户可以读写)
password-db = passwd #(用户信息指定在当前目录的passwd文件)

然后到当前目录下编辑一下passwd文件,形如:

[users]
yyc = 000000

启动服务

openrc

如果使用 openrc,可以直接执行: /etc/init.d/svnserve start

systemd

subversion 没有提供 systemd 的 unit,需要自己写一个。如果像我一样只是临时测试用,可以直接用命令行启动 svnserve:

t-station ~ # svnserve -d -r /var/svn/ --pid-file /var/run/svnd.pid

Check out files & play

svn co svn://localhost/repos --username yyc
]]>
ssh rsa not working 2017-02-21T00:00:00+00:00 http://yangyingchao.github.io/ssh-rsa-not-working Make sure the permissions on the ~/.ssh directory and its contents are proper. When I first set up my ssh key auth, I didn't have the ~/.ssh folder properly set up, and it yelled at me.

Your home directory ~, your ~/.ssh directory and the ~/.ssh/authorized_keys file on the remote machine must be writable only by you: rwx—— and rwxr-xr-x are fine, but rwxrwx— is no good¹, even if you are the only user in your group (if you prefer numeric modes: 700 or 755, not 775).

If ~/.ssh or authorized_keys is a symbolic link, the canonical path (with symbolic links expanded) is checked.

Your ~/.ssh/authorized_keys file (on the remote machine) must be readable (at least 400), but you'll need it to be also writable (600) if you will add any more keys to it.

Your private key file (on the local machine) must be readable and writable only by you: rw-——, i.e. 600.

Also, if SELinux is set to enforcing, you may need to run restorecon -R -v ~/.ssh (see e.g. Ubuntu bug 965663 and Debian bug report #658675; this is patched in CentOS 6).

]]>
strange sigsegv with gdb 2017-01-17T00:00:00+00:00 http://yangyingchao.github.io/strange-sigsegv-with-gdb is this true?

according to: http://stackoverflow.com/questions/13132669/strange-sigsegv-while-calling-java-code-from-c-through-jni

This was indeed what technomage had suggested in the comments. The gdb crash was a red herring because of jvm throwing SIGSEGV which was meant to be handled by jvm.

Once I tell gdb "handle SIGSEGV nostop", it works just fine and I was able to debug my larger program.

]]>
杂记 2016-12-15T00:00:00+00:00 http://yangyingchao.github.io/网络编程杂记

Table of Contents

pipe 缓冲区有多大

网络编程中,有时候会用 pipe 来唤醒 epoll/poll/select ,但 pipe 缓冲区有多大?超了会怎么样?

缓冲区大小与内核版本有关:

  • < 2.6.11 : 与 PAGE_SIZE 相同
  • >= 2.6.11 : 65536 (64K) 字节
  • >= 2.6.35 :仍然为 64KB ,但可以通过 fcntl 来查询和设置。

如果一直写,没有读,写超后, write 要么阻塞,要么失败,与 O_NONBLOCK 的设置有关。

详见: man 7 pipe :

Pipe capacity
  A pipe has a limited capacity.   If the pipe is full, then a
  write(2)  will  block  or  fail, depending  on  whether  the
  O_NONBLOCK   flag    is   set   (see    below).    Different
  implementations have different limits for the pipe capacity.
  Applications should  not rely  on a particular  capacity: an
  application  should be  designed so  that a  reading process
  consumes data as soon as it  is available, so that a writing
  process does not remain blocked.

  In Linux versions before 2.6.11,  the capacity of a pipe was
  the same as the system page size (e.g., 4096 bytes on i386).
  Since Linux 2.6.11, the pipe capacity is 65536 bytes.  Since
  Linux 2.6.35, the default pipe  capacity is 65536 bytes, but
  the  capacity can  be  queried and  set  using the  fcntl(2)
  F_GETPIPE_SZ and F_SETPIPE_SZ  operations.  See fcntl(2) for
  more information.

Socket SO_ERROR vs errno

errno

errno 看起来像是一个全局变量,实则不然:

man 3 errno 显示:

errno is  defined by the ISO  C standard to be  a modifiable
lvalue of  type int,  and must  not be  explicitly declared;
errno may be a macro.   errno is thread-local; setting it in
one thread does not affect its value in any other thread.

SO_ERROR

getsockopt() 提供了另外一种方式来获取 errno: SO_ERRORman 7 socket 显示:

Socket options
  The  socket  options  listed  below  can  be  set  by  using
  setsockopt(2) and  read with  getsockopt(2) with  the socket
  level set  to SOL_SOCKET for all  sockets.  Unless otherwise
  noted, optval is a pointer to an int.

  SO_ERROR
       Get and  clear the  pending socket error.   This socket
       option is read-only.  Expects an integer.

sample:

int errno2(int sfd)
{
  int error = 0;
  socklen_t errlen = sizeof(error);
  if (sfd != -1 && getsockopt(sfd, SOL_SOCKET, SO_ERROR, (void*)&error, &errlen) == 0) {
    ;
  }
  return error;
}

用哪个

答案是:都行。。。。

]]>
Linux:How To Identify Server ECC Memory Modules 2016-07-21T00:00:00+00:00 http://yangyingchao.github.io/Linux:How-To-Identify-Server-ECC-Memory-Modules How do I identify Linux server ECC memory modules from a shell prompt?

You need to count the chips on the module. If memory module has 9 chips per sides, it is ECC memory (EDAC protected memory). It may or may not be registered. If the memory module has 8 chips per side, it is not ECC memory modules.

You can get this information by visiting your BIOS setup menu. Another option is to use dmidecode command to dump bios memory information from a shell prompt:
# dmidecode --type 16

# dmidecode 2.11
SMBIOS 2.5 present.
Handle 0x0016, DMI type 16, 15 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: Single-bit ECC
    Maximum Capacity: 64 GB
    Error Information Handle: Not Provided
    Number Of Devices: 8

To see memory speed and other information, enter:
# dmidecode --type memory

Repost from: http://www.cyberciti.biz/faq/ecc-memory-modules/

]]>
Gentoo Portage 迁移至 git 2016-07-04T00:00:00+00:00 http://yangyingchao.github.io/Gentoo-Portage-迁移至-git 转载自: http://kafiry.lofter.com/post/1d293d40_8016615

更新portage

emerge --update sys-apps/portage

#mkdir /etc/portage/repos.conf
#touch /etc/portage/repos.conf/gentoo.conf
#mv /usr/portage /usr/portage.bak

更改/etc/portage/repos.conf/gentoo.conf

[gentoo]
location = /usr/portage
sync-type = git
sync-uri = git://anongit.gentoo.org/repo/gentoo.git
auto-sync = yes

同步树倒是快。。。

eix:

CACHE_METHOD="/usr/portage parse|ebuild*" eix-update

or

CACHE_METHOD="/usr/portage/ parse|ebuild*"  >>   /etc/eixrc/eixrc

略慢。。。

]]>
LIRS 要点 2016-06-23T00:00:00+00:00 http://yangyingchao.github.io/LIRS-要点

Table of Contents

简单记录一下 LIRS 相关要点。

Recency & Inter-reference Recency

LIRS*: Low Inter-reference Recency Set,中文??

Recency

页面上次访问至今,共访问了多少其他页面。

IRR: Inter-Reference Recency

页最近两次访问之间访问了多少次其他页面。 IRR 越大 (high),表示该页被使用的间隔越长,频率越低,越 “冷”。反之, IRR 越小 (low), 该页使用间隔越小,频率越高,即越 “热”。

冷页与热页

面根据 IRR 分为了两类: LIR (low IRR) 与 HIR (high IRR) ,也即:热页与冷页。中,热页驻留内存,而冷页则不一定驻留。

页的数量略小于 Cache 中能缓存的页面总数大小,再热页限制到达之前,所有页面都是热页。而当热限制到达之后, LIRS 根据算法 来进行后续处理。

Stack & Queue

LIRS* 算法中借用了两个数据结构来实现: Stack & Queue

Stack

tack (栈) 是 LIRS 算法中的主要数据结构,其中存储了所有的热页以及最近使用的冷页。 tack 本身依据 Recency 排序:越老的越靠近栈底,越新的越靠近栈顶。栈里面比 least recent hot page 还的页会从栈中移除 (stack pruning),因此栈底的页始终是热页。

tack 中的冷页其实是住在一个 “试用期”,如果期间冷页被再次引用,则转为热页,否则将最终被栈底移除。

Queue

Queue (队) 中保存了所有驻留内存的冷页。当热页从 Stack 中移除时,该页被加入到 Queue 末尾。而当需要进行页面替代时, LIRS 会从 Queue 的首部取出一个页面以供使用。

页替换规则

LIRS List 中插入页时,会根据待插入的页类型来进行处理:

热页

果待插入页为热页,将该页置于栈顶,而如果该页之前处于栈底,则需对栈进行清理

驻留冷页

先将该页置于栈顶,并做如下处理:

如果该页之前已经在栈里

  1. 则将其转换为热页,并从队中移除。
  2. 将栈底标记为冷页,并移到队尾。
  3. 对栈进行清理

如果该页之前仅在队里(之前为冷页)

不改变冷热状态,但挪动队尾。

非驻留冷页

队首移除一条记录,并将其从 Cache 中驱逐。如果被驱逐的记录之前已经在栈里,则其仍处于未绑定,依然处于试用期。将这个记录与 cache 页绑定,并移植栈顶:果该页之前已经在栈里:

  1. 则将其转换为热页,并从队中移除。
  2. 将栈底标记为冷页,并移到队尾。
  3. 对栈进行清理

如果该页之前仅在队里(之前为冷页)

不改变冷热状态,但挪动队尾。

页的状态

栈里面的页可以是绑定的,页可以是非绑定的,但队里面的页必须是绑定状态的。

]]>
Emacs On OSX 2016-06-22T00:00:00+00:00 http://yangyingchao.github.io/Emacs-On-OSX OS X 下可以通过 Gentoo Prefix 或者 macport 或者 home brew 来安装 Emacs。安装完的 Emacs 使 OS X 原生的 Cocoa UI,很不错。但如果在 Finder 里面直接双机打开文件的话,每次都会新创出一个新的 Frame 来。

在使用 Finder 打开文件时,可不可以这样:

  • 如果没有启动的 Emacs 实例,则创建 Emacs 实例,并打开文件。
  • 如果已经有启动的 Emacs,则在新的 buffer 中打开,并升起 Emacs 窗口。

Google 一下,发现有现成的解决方案:Automator。

具体步骤如下:

  1. 打开 Automator,创建一个新的 App。
  2. 从 Automator 中选择 shell,并添加如下代码:
    #!/bin/bash
    #
    
    EMACS=/opt/Applications/Gentoo/Emacs-24.app/Contents/MacOS/Emacs
    EMACSCLIENT=/opt/usr/bin/emacsclient
    
    export LC_CTYPE=zh_CN.UTF-8
    export LC_ALL=
    
    _is_emacs_daemon_started () {
        netstat -nl 2> /dev/null | awk '{print $NF}' | grep -q "emacs"
    }
    
    _is_emacs_window_exist () {
        _is_emacs_daemon_started && \ $EMACSCLIENT -e '(<= 2 (length (visible-frame-list)))' | grep -q -x t
    }
    
    kill_emacs () {
        if _is_emacs_daemon_started;
        then $EMACSCLIENT -e -n '(let ((desktop-save '\''if-exists)) (dolist (f (cdr-safe (reverse (frame-list)))) (delete-frame f t)) (kill-emacs "yyyyyy"))'
             if _is_emacs_daemon_started;
             then ps -u $UID -o pid,comm= | grep Emacs$ | cut -d' ' -f1 | xargs -n 1 kill
             fi
        fi
        return 0
    }
    
    start_emacs ()
    {
        $EMACS &>/dev/null &
        return $?
    }
    
    main () {
        _is_emacs_daemon_started
        if [ $? -ne 0 ] ; then
            start_emacs
            if [ $? -eq 0 ]; then
                echo ' [sucess]'
            else
                echo ' [faild]'
                return 1
            fi
        fi
    
        if [ "$1" = "-t" ]; then
            $EMACSCLIENT -n "$@"
        elif [ -z "$1" ];  then
            if ! _is_emacs_window_exist;  then
                $EMACSCLIENT -n
            fi
            osascript -e 'tell application "Emacs" to activate'
            return 0
        else
            if ! _is_emacs_window_exist;  then
                $EMACSCLIENT -n
            fi
            if ! echo "$*" | grep -q -e '-n'; then
                osascript -e 'tell application "Emacs" to activate'
            fi
            $EMACSCLIENT -n "$@"
        fi
        osascript -e 'tell application "Emacs" to activate'
    }
    
    main "$@"
    

    其中的 EMACSEMACSCLIENT 根据实际情况填写正确的路径。

    之后将右侧的传递输入改成 “作为自变量”,最后将工程保存为 RunEmacs.App 即可:

    automator-emacs.png

  3. 将 RunEmacs.app 设置为默认程序即可。

PS:

修改 RunEmacs 的图标 http://apple.stackexchange.com/questions/369/can-i-change-the-application-icon-of-an-automator-script%EF%BC%9A

After the script has been created do the following :

  1. Find the source app with the icon you want

  2. Get Info of the source app (cmd-i)

  3. Click on the icon inside the source app's info window (Not the one in the big Preview section at the bottom, if you have one of those; the little one in the top-left corner.)

  4. Copy it (cmd-c)

  5. Get Info of the automator script (i.e., the destination app) (cmd-i)

  6. Click on the icon inside the destination app's info window

  7. Paste the icon from the clipboard (cmd-v)

This method works for every files in Mac OS X.

]]>
2016-06-21T00:00:00+00:00 http://yangyingchao.github.io/疲 貌似很长时间没有在这里写什么东西了,连续两个月的封闭开发,整个人都很疲惫。

等封闭之后,重新开始记录东西了,每周最少写一篇东西,每个季度最少写个总结。

That's it.

]]>