2012年6月20日水曜日

Don't create a sub class of Put

Don't create a sub class of org.apache.hadoop.hbase.client.Put class unless you deliver the class to all HBase region servers. Otherwise, you will see an error message "java.io.IOException: IPC server unable to read call parameters: Error in readFields". What I experienced is described below.

I wrote a sub class of Put because I just wanted to avoid calling Bytes.toBytes(String) for every constructor/method arguments. To be concrete, I wrote the following.

import java.util.List;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.RowLock;
import org.apache.hadoop.hbase.util.Bytes;

public class JPut extends Put
{
    public JPut()
    {
    }

    public JPut(byte[] row)
    {
        super(row);
    }

    public JPut(String row)
    {
        super(toBytes(row));
    }

    public JPut(Put put)
    {
        super(put);
    }

    public JPut(byte[] row, RowLock rowLock)
    {
        super(row, rowLock);
    }

    public JPut(String row, RowLock rowLock)
    {
        super(toBytes(row), rowLock);
    }

    public JPut(byte[] row, long ts)
    {
        super(row, ts);
    }

    public JPut(String row, long ts)
    {
        super(toBytes(row), ts);
    }

    public JPut(byte[] row, long ts, RowLock rowLock)
    {
        super(row, ts, rowLock);
    }

    public JPut(String row, long ts, RowLock rowLock)
    {
        super(toBytes(row), ts, rowLock);
    }

    @Override
    public JPut add(byte[] family, byte[] qualifier, byte[] value)
    {
        return (JPut)super.add(family, qualifier, value);
    }

    public JPut add(String family, String qualifier, byte[] value)
    {
        return (JPut)super.add(toBytes(family), toBytes(qualifier), value);
    }

    @Override
    public JPut add(byte[] family, byte[] qualifier, long ts, byte[] value)
    {
        return (JPut)super.add(family, qualifier, ts, value);
    }

    public JPut add(String family, String qualifier, long ts, byte[] value)
    {
        return (JPut)super.add(toBytes(family), toBytes(qualifier), ts, value);
    }

    public List<KeyValue> get(String family, String qualifier)
    {
        return super.get(toBytes(family), toBytes(qualifier));
    }

    public boolean has(String family, String qualifier, byte[] value)
    {
        return super.has(toBytes(family), toBytes(qualifier), value);
    }

    public boolean has(String family, String qualifier, long ts, byte[] value)
    {
        return super.has(toBytes(family), toBytes(qualifier), ts, value);
    }

    public boolean has(String family, String qualifier, long ts)
    {
        return super.has(toBytes(family), toBytes(qualifier), ts);
    }

    public boolean has(String family, String qualifier)
    {
        return super.has(toBytes(family), toBytes(qualifier));
    }

    private static byte[] toBytes(String string)
    {
        if (string == null)
        {
            return null;
        }
        else
        {
            return Bytes.toBytes(string);
        }
    }
}

Then, I prepared an instance of the class and passed it to HTableInterface.put(Put) method like below.

String rowKey = ...;
String family = ...;
String qualifier = ...;
byte[] value = ...;

JPut jput = new JPut(rowKey);
jput.add(family, qualifier, value);

HTableInterface table = ...;

table.put(jput);

But, this caused "java.io.IOException: IPC server unable to read call parameters: Error in readFields". After grepping the source tree of HBase, I found "Error in readFields" was emitted by HbaseObjectWritable.java that runs on the client side. However, on the other hand, "IPC server unable to read call parameters" was found in HBaseServer.java that runs on the server side. So, I checked a log file of an HBase region server and found a stack trace like below.

2012-06-19 22:29:39,930 WARN org.apache.hadoop.ipc.HBaseServer: Unable to read call parameters for client 192.168.62.133
java.io.IOException: Error in readFields
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:655)
        at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1248)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1177)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:713)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:505)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:480)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error in readFields
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:655)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:565)
        at org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:652)
        ... 9 more
Caused by: java.io.IOException: Can't find class JPut
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:644)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:565)
        at org.apache.hadoop.hbase.client.Action.readFields(Action.java:101)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:652)
        ... 12 more
Caused by: java.lang.ClassNotFoundException: JPut
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:699)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:641)
        ... 15 more

The log said that the class 'JPut' was not found. I went back to the source code 'HBaseServer.java' and confirmed that it does reflection.

Writable param;
try {
  param = ReflectionUtils.newInstance(paramClass, conf);//read param
  param.readFields(dis);
} catch (Throwable t) {
  LOG.warn("Unable to read call parameters for client " +
           getHostAddress(), t);
  final Call readParamsFailedCall =
    new Call(id, null, this, responder, callSize);
  ByteArrayOutputStream responseBuffer = new ByteArrayOutputStream();

  setupResponse(responseBuffer, readParamsFailedCall, Status.FATAL, null,
      t.getClass().getName(),
      "IPC server unable to read call parameters: " + t.getMessage());
  responder.doRespond(readParamsFailedCall);
  return;
}

Finally I understood that I should not have used a sub class of 'Put'.

2012年6月18日月曜日

Country code enum in Java (ISO 3166-1 alpha-2/alpha-3/numeric)

Edit on 2013-Jan-30
nv-i18n (https://github.com/TakahikoKawasaki/nv-i18n) supersedes CountryCode.


A Java enum listing up ISO 3166-1 country codes is available as open source (Apache License version 2.0) at GitHub.

CountryCode
https://github.com/TakahikoKawasaki/CountryCode

Below is the description of the GitHub project.


CountryCode

ISO 3166-1 (alpha-2/alpha-3/numeric) country code enum in Java.

License

Apache License, Version 2.0

Download

git clone git://github.com/TakahikoKawasaki/CountryCode.git

Javadoc

CountryCode Javadoc

Example

CountryCode cc = CountryCode.getByCode("JP");

System.out.println("Country name = " + cc.getName());  // "Japan"
System.out.println("ISO 3166-1 alpha-2 code = " + cc.getAlpha2());  // "JP"
System.out.println("ISO 3166-1 alpha-3 code = " + cc.getAlpha3());  // "JPN"
System.out.println("ISO 3166-1 numeric code = " + cc.getNumeric());  // 392




国コード Java enum (ISO 3166-1 alpha-2/alpha-3/numeric)

2013 年 1 月 30 日編集
nv-i18n (https://github.com/TakahikoKawasaki/nv-i18n) が CountryCode の後継となります。

ISO 3166-1 国コードを列挙する Java enum が、オープンソース (Apache License version 2.0) として GitHub で公開されています。


CountryCode
https://github.com/TakahikoKawasaki/CountryCode

その GitHub プロジェクトの説明は次のとおりです。


CountryCode

ISO 3166-1 (alpha-2/alpha-3/numeric) 国コードを表す Java enum.

ライセンス

Apache License, Version 2.0

ダウンロード

git clone git://github.com/TakahikoKawasaki/CountryCode.git

Javadoc

CountryCode Javadoc

CountryCode cc = CountryCode.getByCode("JP");

System.out.println("国名 = " + cc.getName());  // "Japan"
System.out.println("ISO 3166-1 alpha-2 コード = " + cc.getAlpha2());  // "JP"
System.out.println("ISO 3166-1 alpha-3 コード = " + cc.getAlpha3());  // "JPN"
System.out.println("ISO 3166-1 numeric コード = " + cc.getNumeric());  // 392



2012年6月15日金曜日

git pushでHTTP 411エラーが出たときの対処法

git push をおこなったとき、サーバーから HTTP ステータスコード 411 が返され、失敗してしまうことがある。

[図1] TortoiseGit でのエラー表示例

[図2] Eclipse でのエラー表示例

HTTP ステータスコード 411 の定義には「The server refuses to accept the request without a defined Content- Length. The client MAY repeat the request if it adds a valid Content-Length header field containing the length of the message-body in the request message.」と書かれている。つまり、このステータスコードは、クライアントの HTTP リクエストに Content-Length ヘッダーが付いていないことを理由としてサーバーがリクエストを拒否する場合に返されるのであるが、git push のときにこのステータスコードが返される原因は、push しようとしているデータ量が多過ぎることにある。


この問題の対処法は、git の設定値 http.postBuffer の値を大きくすることである。例えばこの値を 100MB (100 * 1024 * 1024 = 104857600) にするには、コマンドラインでは次のように打ち込む。

git config http.postBuffer 104857600

TortoiseGit の GUIでおこなうには、エクスプローラのコンテキストメニューから、TortoiseGit → Settings で TortoiseGit の設定ダイアログを表示し、Git → Config → Edit local git/config で設定ファイルを開き、http.postBuffer に関する設定を追記する。

[http]
  postBuffer = 104857600

[図3] git 設定ファイルに http.postBuffer を追記