Feature

The performance measurement result of K2HFTFUSE


Contents

Performance measurement result

Comparison with other products Measurement results

Summary

Appendix


Performance measurement result

Performance result (no bottleneck environment)

Transfer amount was measured in an environment without hardware bottleneck. We used 10 Gbps network and SSD.

Measured content

We prepared three patterns of data files.

Figure3

Hardware used

Environment used

Measuring method

Data input method

Data aggregation method

Performance measurement result

Figure1

The maximum transfer volume is 250 MB. (When sending 4096 Byte record)

Record length Average Transferred Records (Seconds) Average transfer bytes (seconds)
10 Byte 937,779 8.9 MB/S
1024 Byte 219,578 214.4 MB/S
4096 Byte 64,943 253.7 MB/S

Measurement in 1 Gbps network (network bottleneck environment)

This is the result when the test aimed at measuring the transfer amount is carried out in 1 Gbps network as above. This result behaves when network performance is insufficient.

Hardware used

Measuring method

For the measurement of the transfer amount, we calculated the number by counting the number of lines of the file aggregated every 10 seconds. When the measurement is over, the file is deleted, and the next data is written to the re-created file.

Performance measurement result

Figure7

The transfer amount was the maximum at about 100 MB and the difference was not seen much between the long size and the middle size.

As a result of checking with the monitoring tool, we were confirmed that the upper limit of the line speed has actually reached. Therefore, in this measurement, we can see the behavior at the upper limit of the network, but the original performance can not be measured.

Summary


Measurement in HDD use environment (bottleneck environment of file / device)

As in the above case, it is the result when the test aiming at the transfer amount measurement was carried out by HDD, not SSD. This result behaves when the performance of the file device is insufficient.

Hardware used

Measuring method

We checked the file size every 10 seconds and calculated the transfer amount from the difference from the size before 10 seconds ago.

Performance measurement result

Figure11

The transfer amount became the maximum at around 150 MB, and the difference between the long size and the middle size was not seen much. There was a phenomenon that the performance suddenly decreased when further transfer was continued after measurement. So, the graph below shows the transition to completion of transfer.

Figure4

We had become obvious by using the monitoring tool at the same time with the measurement. Free memory was allocated to the cache and gradually decreased, and it was late when it was exhausted. At the time of this graph, the fastest speed did not come out around 150 seconds, and in the vicinity of 700 seconds, the free memory was almost gone.

Since this measurement uses HDD instead of SSD, it can be imagined that the speed of the disk can not keep up with writing data and eventually exhausted memory. We changed the output destination to /dev/null and looked at the usage status of the network bandwidth, the transition did not slow down halfway through the transfer. For this reason We could confirm that the HDD is a bottleneck.

Summary


Comparison with other products Measurement results

We tried to compare with fluentd and kafka how to use other products in the same use case.
This is the product and version which measured the performance this time.

Data input method

Data aggregation method

Performance measurement result


10 Byte records
Figure11


1024 Byte records
Figure12


4096 Byte records

Figure13


Comparison with other products Measurement results Summary


Summary

On the performance of K2HFTFUSE

The performance of K2HFTFUSE has reached the limit until 10 Gbps network and high-speed storage are prepared. When using it in the environment below that, the network and disk writing speed reaches the upper limit before the limit of K2HFTFUSE. Therefore, it can be used effectively without concern about the limit of K2HFTFUSE on the majority of existing systems.

Effective usecase with K2HFTFUSE

This time, we investigated the performance mainly for log transfer which is one of the assumed use cases of K2HFTFUSE. As a result, we were able to confirm performance beyond our expectation in log transfer.

From this result, We believe the following usage examples are effective.

Moreover, We think that there are many scenes that can take advantage of the following features when compared with other products.

in conclusion

We compared this with fluentd and kafka which have similar functions as comparison objects, but We are not a professional of these two products. Therefore, We think that better results will be obtained if you are familiar with these products tuning and measuring. If there is a result of tuning with the same use case We think that you should publish it by all means. Also, if there are measurement results in other use cases, We would like to try measurement even with K2HFTFUSE.


Appendix

This is the measurement method and the setting file used for measurement.


About determining how to measure

We will briefly introduce preparation and trial and error to the measuring method.

Installation

We arranged the development server and our network within our company, installed each product and checked the communication in the tutorial.
We use fluentd and kafka without tune for speeding up especially.

Preliminary test

We measured while changing data file and server configuration.

Based on this result, we examined the measurement method and data size when testing in a dedicated environment.


Measurement script

For files aggregated on the server side, the number of bytes to be written was calculated using the following script.

#!/bin/bash
pre=0
while :
do
	date=`date`
	now=`ls -l log.txt | awk '{print $5;}'`
	num=`expr $now - $pre`

	echo -n "$date "
	echo $num

	pre=$now
	sleep 10
done

K2HFTFUSE setting file (receiving side)

This is the setting file used for this measurement.

# k2hftfuse server side
#
# GLOBAL SECTION
#
[GLOBAL]
FILEVERSION			= 1
DATE				= Thu, 03 Sep 2015 17:27:28 +0900
GROUP				= K2HFUSETEST
MODE				= SERVER
DELIVERMODE			= random
MAXCHMPX			= 5
REPLICA				= 0
MAXMQSERVER			= 64
MAXMQCLIENT			= 64
MQPERATTACH			= 1
MAXQPERSERVERMQ			= 2
MAXQPERCLIENTMQ			= 32
MAXMQPERCLIENT			= 8
MAXHISTLOG			= 0
PORT				= 8020
CTLPORT				= 8021
SELFCTLPORT			= 8021
RWTIMEOUT			= 10000
RETRYCNT			= 1000
CONTIMEOUT			= 500000
MQRWTIMEOUT			= 20000
MQRETRYCNT			= 2000
MQACK				= no
DOMERGE				= on
AUTOMERGE			= on
MERGETIMEOUT			= 0
SOCKTHREADCNT			= 8
MQTHREADCNT			= 4
MAXSOCKPOOL			= 8
SOCKPOOLTIMEOUT			= 0
SSL				= no
K2HFULLMAP			= on
K2HMASKBIT			= 8
K2HCMASKBIT			= 7
K2HMAXELE			= 32
K2HPAGESIZE			= 5000

#
# SERVER NODES SECTION
#
[SVRNODE]
NAME				= xxxx.yahoo.co.jp
SSL				= no

#
# SLAVE NODES SECTION
#
[SLVNODE]
NAME				= [.]*
CTLPORT				= 8022

[K2HFTFUSESVR]
TYPE				= file
FILE_BASEDIR			= /tmp/k2hftfusesvr
FILE_TIMEFORM			= "%F %T.%-"
#FORMAT				= "%T +0900 temotest': {'message':'%l','host':'%H'}\n" # for JSON format
FORMAT				= "%L"
FILE_UNIFY			= log/unify.log

#
# VIM modelines
#
# vim:set ts=4 fenc=utf-8:
#

K2HFTFUSE setting file (sending side)

This is the setting file used for this measurement.

# k2hftfuse slave side
#
# GLOBAL SECTION
#
[GLOBAL]
FILEVERSION			= 1
DATE				= Thu, 03 Sep 2015 17:27:28 +0900
GROUP				= K2HFUSETEST
MODE				= SLAVE
DELIVERMODE			= random
MAXCHMPX			= 5
REPLICA				= 0
MAXMQSERVER			= 64
MAXMQCLIENT			= 32
MQPERATTACH			= 8
MAXQPERSERVERMQ			= 16
MAXQPERCLIENTMQ			= 1
MAXMQPERCLIENT			= 8
MAXHISTLOG			= 1000
PORT				= 8020
CTLPORT				= 8022
SELFCTLPORT			= 8022
RWTIMEOUT			= 100000
RETRYCNT			= 10000
MQRWTIMEOUT			= 50
MQRETRYCNT			= 20000
CONTIMEOUT			= 500000
MQACK				= no
DOMERGE				= on
AUTOMERGE			= on
MERGETIMEOUT			= 0
SOCKTHREADCNT			= 0
MQTHREADCNT			= 8
MAXSOCKPOOL			= 8
SOCKPOOLTIMEOUT			= 0
SSL				= no
K2HFULLMAP			= on
K2HMASKBIT			= 8
K2HCMASKBIT			= 4
K2HMAXELE			= 32

#
# SERVER NODES SECTION
#
[SVRNODE]
NAME				= xxxx.yahoo.co.jp
PORT				= 8020
CTLPORT				= 8021
SSL				= no

#
# SLAVE NODES SECTION
#
[SLVNODE]
NAME				= [.]*
CTLPORT				= 8022

#
# K2HTPDTOR
#
[K2HTPDTOR]
K2HTPDTOR_BROADCAST		= no
K2HTPDTOR_FILTER_TYPE		= DELKEY

[K2HFTFUSE]
K2HTYPE				= mem
K2HMASKBIT			= 8
K2HCMASKBIT			= 4
K2HMAXELE			= 32
K2HPAGESIZE			= 4096
DTORTHREADCNT			= 2
BINTRANS			= no
TRANSLINECNT			= 10000
TRANSTIMEUP			= 10
BYTELIMIT			= 0

#
# K2HFTFUSE_RULE_DIR( K2HFTFUSE sub rule )
#

[K2HFTFUSE_RULE_DIR]
TARGET          = senddir
TRUNS           = on
DEFAULTALL      = ALLOW

#
# VIM modelines
#
# vim:set ts=4 fenc=utf-8:
#
Feature