Details TOP Build

Usage

This is an explanation about K2HFTFUSE installation and easy operation confirmation.

1. Creating a usage environment

Install K2HFTFUSE

There are two ways to install K2HFTFUSE in your environment.
One is to download and install the package of K2HFTFUSE from packagecloud.io.
The other way is to build and install K2HFTFUSE from source code yourself.
These methods are described below.

Installing packages

The K2HFTFUSE publishes packages in packagecloud.io - AntPickax stable repository so that anyone can use it.
The package of the K2HFTFUSE is released in the form of Debian package, RPM package.
Since the installation method differs depending on your OS, please check the following procedure and install it.

Debian(Stretch) / Ubuntu(Bionic Beaver)

$ sudo apt-get update -y
$ sudo apt-get install curl -y
$ curl -s https://packagecloud.io/install/repositories/antpickax/stable/script.deb.sh | sudo bash
$ sudo apt-get install k2hftfuse

To install the developer package, please install the following package.

$ sudo apt-get install k2hftfuse-dev

Fedora28 / CentOS7.x(6.x)

$ sudo yum makecache
$ sudo yum install curl -y
$ curl -s https://packagecloud.io/install/repositories/antpickax/stable/script.rpm.sh | sudo bash
$ sudo yum install k2hftfuse

To install the developer package, please install the following package.

$ sudo yum install k2hftfuse-devel

Other OS

If you are not using the above OS, packages are not prepared and can not be installed directly.
In this case, build from the source code described below and install it.

OS preference

k2HFTFUSE is one of FUSE (Filesystem in Userspace) client application.
Therefore, you need to set up the system(OS) before using K2HFTFUSE.
An example of the setting is shown below. For details, refer to FUSE (Filesystem in Userspace).

/etc/fuse.conf

To use K2HFTFUSE other than root, write the following setting in this file.

user_allow_other

/dev/fuse

To use K2HFTFUSE other than root, you need to set read/write permission to this device file.

$ sudo chmod -f a+rw /dev/fuse

/bin/fusermount

To use K2HFTFUSE other than root, you need to set execute permission to this file.

$ sudo chmod -f o+rx /bin/fusermount

2. Operation check

Sample Configuration

The following is the configuration used for K2HFTFUSE and K2HFTFUSESVR test. please refer.

Case not relaying

Source host

K2HFTFUSE
CHMPX

It is the same configuration as K2HFTFUSE, and you need to start CHMPX before K2HFTFUSE starting.

Terminating host

K2HFTFUSESVR
CHMPX

It is the same configuration as K2HFTFUSESVR, and you need to start CHMPX before K2HFTFUSESVR starting.

Case relaying

Source host

K2HFTFUSE
CHMPX

It is the same configuration as K2HFTFUSE, and you need to start CHMPX before K2HFTFUSE starting.

Relay host

K2HFTFUSESVR
CHMPX server node to connect from source host

It is the same configuration as K2HFTFUSESVR, and you need to start CHMPX before K2HFTFUSESVR starting.

CHMPX slave node to connect to terminating host

Terminating host

K2HFTFUSESVR
CHMPX

It is the same configuration as K2HFTFUSESVR, and you need to start CHMPX before K2HFTFUSESVR starting.

Operation check

I will try to transfer the contents of the actual log file.
In this example, we will transfer the Apache log file with K2HFTFUSE.

Precondition

10 hosts(foo[0-9].yahoo.co.jp) provide web services in the Apache program.
The access.log and error.log output by the Apache program are output to the /home/apache/logs directory of each host.
Files other than these are not output to this directory.
And we will aggregate log files on one server bar.yahoo.co.jp.

Configuration

#
# CHMPX GLOBAL SETTING
#
GLOBAL:
    {
        FILEVERSION:            1,
        DATE:                   "Thu, 01 Dec 2016 00:00:00 +0900",
        GROUP:                  K2HFUSETEST,
        MODE:                   SLAVE,
        DELIVERMODE:            random,
        MAXCHMPX:               32,
        REPLICA:                0,
        MAXMQSERVER:            8,
        MAXMQCLIENT:            8,
        MQPERATTACH:            1,
        MAXQPERSERVERMQ:        2,
        MAXQPERCLIENTMQ:        8,
        MAXMQPERCLIENT:         4,
        PORT:                   18020,
        CTLPORT:                18022,
        SELFCTLPORT:            18022,
        RWTIMEOUT:              100000,
        RETRYCNT:               1000,
        CONTIMEOUT:             500000,
        MQRWTIMEOUT:            50,
        MQRETRYCNT:             100000,
        DOMERGE:                on,
        SSL:                    no,
        K2HFULLMAP:             on,
        K2HMASKBIT:             4,
        K2HCMASKBIT:            4,
        K2HMAXELE:              2
    }

#
# CHMPX SERVER NODES SECTION
#
SVRNODE:
    [
        {
            NAME:               foo[0-9].yahoo.co.jp,
            PORT:               8020,
            CTLPORT:            8021,
            SSL:                no
        }
    ]

#
# CHMPX SLAVE NODES SECTION
#
SLVNODE:
    [
        {
            NAME:               "[.]*",
            CTLPORT:            8022
        }
    ]

#
# K2HTPDTOR(K2HASH TRANSACTION PLUGIN)
#
K2HTPDTOR:
    {
        K2HTPDTOR_BROADCAST:    no,
    }

#
# K2HFTFUSE
#
# K2HTYPE       type of k2hash used by transfer mode
# K2HFILE       file path of k2hash used by transfer mode, when file type
# K2HFULLMAP    mapping type of k2hash used by transfer mode, when file type
# K2HINIT       initializing of k2hash used by transfer mode, when file type
# K2HMASKBIT    init mask bit count of k2hash used by transfer mode, when file type
# K2HCMASKBIT   collision mask bit count of k2hash used by transfer mode, when file type
# K2HMAXELE     maximum element count of k2hash used by transfer mode, when file type
# K2HPAGESIZE   page size of k2hash used by transfer mode, when file type
# DTORTHREADCNT k2hdtor thread count
# DTORCTP       custom k2hdtor plugin file name(or path)
# BINTRANS      transfer as binary data array
# EXPIRE        grant the expiration date to transfer
# TRANSLINECNT  transfer line count limit at one time(default 0)
# TRANSTIMEUP   transfer timeup limit(default 0)
# BYTELIMIT     maximum bytes for one data length(default 0, means no limit)
#
K2HFTFUSE:
    {
        K2HTYPE:                mem,
        K2HMASKBIT:             4,
        K2HCMASKBIT:            4,
        K2HMAXELE:              2,

        #
        # K2HFTFUSE_RULE_DIR( K2HFTFUSE sub rule )
        #
        # TARGET        target directory path
        # TRUNS         enable/disable flag for transfer
        # OUTPUTFILE    enable/disable flag for put file
        # PLUGIN        plugin program path
        # DEFAULTALL    default rule as DENY or ALLOW(DENY as default)
        # ALLOW         allowed rule, rule is static string or regex. and convert rule when regex.
        # DENY          denied rule, rule is static string or regex. and convert rule when regex.
        #
        K2HFTFUSE_RULE_DIR:
        [
            {
                TARGET:             /,
                TRUNS:              off,
                OUTPUTFILE:         /dev/null,
                DEFAULTALL:         ALLOW
            },
        ],

        #
        # K2HFTFUSE_RULE( K2HFTFUSE sub rule )
        #
        # TARGET        target file path
        # TRUNS         enable/disable flag for transfer
        # OUTPUTFILE    enable/disable flag for put file
        # PLUGIN        plugin program path
        # DEFAULTALL    default rule as DENY or ALLOW
        # ALLOW         allowed rule, rule is static string or regex. and convert rule when regex.
        # DENY          denied rule, rule is static string or regex. and convert rule when regex.
        #
        K2HFTFUSE_RULE:
        [
            {
                TARGET:             access.log,
                TRUNS:              on,
                OUTPUTFILE:         /dev/null,
                DEFAULTALL:         ALLOW
            },
            {
                TARGET:             error.log,
                TRUNS:              on,
                OUTPUTFILE:         /dev/null,
                DEFAULTALL:         ALLOW
            },
        ]
    }
#
# CHMPX GLOBAL SETTING
#
GLOBAL:
    {
        FILEVERSION:            1,
        DATE:                   "Thu, 01 Dec 2016 00:00:00 +0900",
        GROUP:                  K2HFUSETEST,
        MODE:                   SERVER,
        DELIVERMODE:            random,
        MAXCHMPX:               32,
        REPLICA:                0,
        MAXMQSERVER:            8,
        MAXMQCLIENT:            8,
        MQPERATTACH:            1,
        MAXQPERSERVERMQ:        2,
        MAXQPERCLIENTMQ:        8,
        MAXMQPERCLIENT:         4,
        PORT:                   18020,
        CTLPORT:                18021,
        SELFCTLPORT:            18021,
        RWTIMEOUT:              100000,
        RETRYCNT:               1000,
        CONTIMEOUT:             500000,
        MQRWTIMEOUT:            50,
        MQRETRYCNT:             100000,
        DOMERGE:                on,
        SSL:                    no,
        K2HFULLMAP:             on,
        K2HMASKBIT:             4,
        K2HCMASKBIT:            4,
        K2HMAXELE:              2
    }

#
# CHMPX SERVER NODES SECTION
#
SVRNODE:
    [
        {
            NAME:               foo[0-9].yahoo.co.jp,
            SSL:                no
        }
    ]

#
# CHMPX SLAVE NODES SECTION
#
SLVNODE:
    [
        {
            NAME:               "[.]*",
            CTLPORT:            8022
        }
    ]

#
# K2HFTFUSESVR
#
# TYPE              { trans | file | both }
# FILE_BASEDIR      output file base directory
# FILE_UNIFY        if puts one file for all, specify relative file path.(only file type)
# FILE_TIMEFORM     format for time data like strftime function.(only file type) + '%-': ns(decimal), %%=%
# PLUGIN            program file path for plugin
# FORMAT            output format.(only file type)
#                       %H  hostname
#                       %P  thread(process) id
#                       %F  file path
#                       %f  file name
#                       %T  time, formatted by FILE_TIMEFORM
#                       %L  output content data
#                   * If FORMAT is not specified, it means "%L".
#
# TRANSCONF         slave chmpx configuration file path.(only trans type)
# K2HTYPE           type of k2hash used by transfer mode
# K2HFILE           file path of k2hash used by transfer mode, when file type
# K2HFULLMAP        mapping type of k2hash used by transfer mode, when file type
# K2HINIT           initializing of k2hash used by transfer mode, when file type
# K2HMASKBIT        init mask bit count of k2hash used by transfer mode, when file type
# K2HCMASKBIT       collision mask bit count of k2hash used by transfer mode, when file type
# K2HMAXELE         maximum element count of k2hash used by transfer mode, when file type
# K2HPAGESIZE       page size of k2hash used by transfer mode, when file type
# DTORTHREADCNT     k2htpdtor thread count
# DTORCTP           custom transaction plugin
#
K2HFTFUSESVR:
    {
        TYPE:                   file,
        FILE_BASEDIR:           /home/apache/all,
        FILE_UNIFY:             log/unify.log,
        FILE_TIMEFORM:          "%F %T(%s %-)",
        #PLUGIN:                 /usr/local/bin/myprogram,
        FORMAT:                 "%H:%F(%P):%f[%T] %L",
        K2HMASKBIT:             4,
        K2HCMASKBIT:            4,
        K2HMAXELE:              2
    }

Test started

First, prepare the hosts that aggregate the logs.

Next, start the programs on the host on which Apache is running.

Check mount point normally

Start Apache program

Start up Apache, output the logs, and check the files to be aggregated on the aggregation server.

Run other program instead of Apache

Even without running Apache, you can write to the file on the sending host as follows.
In the following example, “TEST STRING” is transferred to the aggregated log file.

In this way it can be confirmed that the file contents can be transferred.

The advantage of K2HFTFUSE is that there is no need to change the transfer source (program).
And aggregation server and route can be determined by setting, and secure and high speed file / message transfer is possible.
In addition, you can start your external program on the transfer destination server, making your own processing very easy.
Please change the setting file and try it out.

Details TOP Build