incubator-crail.git
11 days agoCheckstyle: rename style file master
Jonas Pfefferle [Fri, 7 Sep 2018 14:06:37 +0000 (16:06 +0200)] 
Checkstyle: rename style file

Rename file style.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
11 days agoCheckstyle plugin
Jonas Pfefferle [Fri, 7 Sep 2018 12:49:22 +0000 (14:49 +0200)] 
Checkstyle plugin

Move checkstyle plugin to parent pom.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
11 days agoCheckstyle plugin with google style
Jonas Pfefferle [Wed, 28 Mar 2018 20:54:19 +0000 (22:54 +0200)] 
Checkstyle plugin with google style

Add checkstyle plugin but do not check on compilation yet.
Checkstyle can be run with 'mvn checkstyle:check'

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-19

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoApache parent pom
Jonas Pfefferle [Tue, 21 Aug 2018 09:36:53 +0000 (11:36 +0200)] 
Apache parent pom

Use Apache parent pom. It already includes all necessary pieces
to do source and binary releases and push to maven central.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-34

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoDocumentation: remove logo
Jonas Pfefferle [Tue, 4 Sep 2018 09:01:56 +0000 (11:01 +0200)] 
Documentation: remove logo

Remove Crail logo from documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoDocumentation: Spark-IO
Jonas Pfefferle [Wed, 15 Aug 2018 09:15:36 +0000 (11:15 +0200)] 
Documentation: Spark-IO

Add Spark-IO documentation: building and configuration.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoDocumentation: HDFS Adapter
Jonas Pfefferle [Wed, 15 Aug 2018 08:45:54 +0000 (10:45 +0200)] 
Documentation: HDFS Adapter

Add Spark HDFS Adapater documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoDocumentation: iobench & fsck
Jonas Pfefferle [Tue, 14 Aug 2018 09:26:08 +0000 (11:26 +0200)] 
Documentation: iobench & fsck

Add iobench and fsck documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
12 days agoDocumentation: readthedocs Documentation
Jonas Pfefferle [Fri, 10 Aug 2018 08:25:01 +0000 (10:25 +0200)] 
Documentation: readthedocs Documentation

New Documentation using readthedocs based on github README.
Extended and adapted to latest Crail version.
Will be hosted at http://incubator-crail.readthedocs.io

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 weeks agoAdjusting to latest DiSNI/DaRPC (v1.7)
Patrick Stuedi [Tue, 4 Sep 2018 14:32:54 +0000 (16:32 +0200)] 
Adjusting to latest DiSNI/DaRPC (v1.7)

https://issues.apache.org/jira/browse/CRAIL-50

Signed-off-by: Patrick Stuedi <pstuedi@apache.com>
5 weeks agoDocker: fix cache hugepage requirement
Jonas Pfefferle [Fri, 3 Aug 2018 08:36:57 +0000 (10:36 +0200)] 
Docker: fix cache hugepage requirement

Allow users to use the Crail shell without providing
hugepages via a volume mount to Docker.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 weeks agoDocker build: TCP and RDMA tier
Jonas Pfefferle [Thu, 26 Jul 2018 11:35:25 +0000 (13:35 +0200)] 
Docker build: TCP and RDMA tier

Docker build files for the TCP and RDMA tier.
User can extend configuration to use NVMf or other tiers.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-49

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 months agoNVMf: new jNVMf version 1.4
Jonas Pfefferle [Thu, 5 Jul 2018 11:43:52 +0000 (13:43 +0200)] 
NVMf: new jNVMf version 1.4

Use new jNVMf to fix dependency issue with DiSNI 1.6.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 months agoThrow more meaningful exception if datapath is invalid or has no write
Patrick Stuedi [Tue, 26 Jun 2018 08:43:04 +0000 (10:43 +0200)] 
Throw more meaningful exception if datapath is invalid or has no write
permissions.

Fixes the issue of invalid datapath or missing write permissions on the
datapath for both the RDMA and the TCP storage tier.

https://issues.apache.org/jira/browse/CRAIL-45

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoMake sure statistics are updated across all classes in the Crail HDFS
Patrick Stuedi [Mon, 25 Jun 2018 11:46:45 +0000 (13:46 +0200)] 
Make sure statistics are updated across all classes in the Crail HDFS
module

Previously statistics where only captured in HDFSOutputStream and parts
of the CrailHadoopFileSystem.

https://issues.apache.org/jira/browse/CRAIL-26

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoDelegate implementation of CrailHDFS to CrailHadoopFileSystem
Patrick Stuedi [Mon, 25 Jun 2018 11:16:01 +0000 (13:16 +0200)] 
Delegate implementation of CrailHDFS to CrailHadoopFileSystem

Implement AbstractFileSystem by delegating function calls to
CrailHadoopFileSystm. The same can be done also by using
DelegateToFileSystem from hadoop-common, but this results in additional
operations (e.g., exists(path) and checkPath(path)).

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-27

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoAdding extra config parameters to control backlog and connnection
Patrick Stuedi [Fri, 22 Jun 2018 09:37:51 +0000 (11:37 +0200)] 
Adding extra config parameters to control backlog and connnection
timeouts for RDMA-based RPC and storage backends

The backlog parameter controlls the backlog of incoming connection
requests for rdma storage and rdma RPC. The connectiontimeout puts a
limit on the time to wait for the connection establishement event.

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoAdjustments to comply with DiSNI v1.6 and DaRPC v1.6
Patrick Stuedi [Thu, 21 Jun 2018 14:40:26 +0000 (16:40 +0200)] 
Adjustments to comply with DiSNI v1.6 and DaRPC v1.6

The connect/bind API of DiSNI version 1.6 has changed, affecting
rpc-darpc and storage-rdma.

https://issues.apache.org/jira/browse/CRAIL-41

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoScripts: fix log/pid file for multiple datanodes
Jonas Pfefferle [Wed, 20 Jun 2018 13:45:47 +0000 (15:45 +0200)] 
Scripts: fix log/pid file for multiple datanodes

Fix using the same filename when starting multiple storage
tiers on the same node. Append arguments from slave file
to make path unique.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-29

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 months agoScripts: add crail-env.sh
Jonas Pfefferle [Wed, 20 Jun 2018 14:52:46 +0000 (16:52 +0200)] 
Scripts: add crail-env.sh

Add crail-env.sh file to allow setting parameters for
the start scripts like extra java options.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 months agoStart script: remove heap size argument
Jonas Pfefferle [Wed, 20 Jun 2018 11:36:11 +0000 (13:36 +0200)] 
Start script: remove heap size argument

Remove Xmx and MaxDirectMemorySize argument from start script
since it does not allow to run Crail on machines with less
than 64G.

https://issues.apache.org/jira/browse/CRAIL-40

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
2 months agoCheck for already existing cache directory with the same ID
Patrick Stuedi [Wed, 20 Jun 2018 13:06:02 +0000 (15:06 +0200)] 
Check for already existing cache directory with the same ID

Throw an exception if a cache directory for the same ID
(reflecting a particular instance of CrailStore) already exists

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
2 months agoIntroducing StorageUtils as a common ground for storage utility
Patrick Stuedi [Wed, 20 Jun 2018 12:11:04 +0000 (14:11 +0200)] 
Introducing StorageUtils as a common ground for storage utility
functions

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
3 months agoMake default configuration easier to run
Patrick Stuedi [Tue, 19 Jun 2018 13:21:07 +0000 (15:21 +0200)] 
Make default configuration easier to run

Better default values for cachepath and tcp storage path. Throw an
exception if paths are invalid. Throw an exception if interface is
invalid. Trim down crail-site.conf.template to only include the
necessary parameters.

https://issues.apache.org/jira/browse/CRAIL-15

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 months agoMove release date in HISTORY v1.0
Jonas Pfefferle [Tue, 15 May 2018 11:04:26 +0000 (13:04 +0200)] 
Move release date in HISTORY

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoApache incubator disclaimer
Jonas Pfefferle [Mon, 14 May 2018 11:02:57 +0000 (13:02 +0200)] 
Apache incubator disclaimer

Add Aapache incubator disclaimer.

https://issues.apache.org/jira/browse/CRAIL-33

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoMove IBM copyright NOTICE to CREDITS file
Jonas Pfefferle [Mon, 14 May 2018 10:59:28 +0000 (12:59 +0200)] 
Move IBM copyright NOTICE to CREDITS file

Move the IBM copyright notice from the NOTICE file to the CREDITS file.

https://issues.apache.org/jira/browse/CRAIL-33

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoNVMf: decrease keep alive interval
Jonas Pfefferle [Thu, 3 May 2018 11:37:44 +0000 (13:37 +0200)] 
NVMf: decrease keep alive interval

Set keep alive interval to lower value to not run into the
situation where we missed the keep alive deadline and the connection
is shutdown by the controller.

https://issues.apache.org/jira/browse/CRAIL-32

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoNVMf: fix staging buffer cache get race condition
Jonas Pfefferle [Mon, 7 May 2018 07:16:22 +0000 (09:16 +0200)] 
NVMf: fix staging buffer cache get race condition

Fixes NVMf staging buffer cache get. Do not throw an exception
if we can't free the previous buffer in the hashmap. If we can't
free it it was already freed by allocateFreeBuffers.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoNVMf: staging buffer cache clear buffer
Jonas Pfefferle [Thu, 3 May 2018 11:38:00 +0000 (13:38 +0200)] 
NVMf: staging buffer cache clear buffer

Clear buffer position and limit when getting the buffer of the
free queue since position and limit might have been modified before.

https://issues.apache.org/jira/browse/CRAIL-31

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoNVMf: staging buffer cache existing entry bug
Jonas Pfefferle [Wed, 2 May 2018 10:53:47 +0000 (12:53 +0200)] 
NVMf: staging buffer cache existing entry bug

A buffer might already exist at a certain remote address (resp. LBA) if
a file was deleted and a block was reassigned to a different file. Do not
throw an exception but free the buffer. If we can't free it something
went wrong as there should never be two writers writing to the same
remote address.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-30

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoREADME: give all three options to download src
Jonas Pfefferle [Mon, 23 Apr 2018 10:54:26 +0000 (12:54 +0200)] 
README: give all three options to download src

Include all three options to download the Crail source:
Apache, github and Apache source distro.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoMove HISTORY release date
Jonas Pfefferle [Mon, 23 Apr 2018 09:21:00 +0000 (11:21 +0200)] 
Move HISTORY release date

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoREADME: build from source distro
Jonas Pfefferle [Mon, 23 Apr 2018 09:19:27 +0000 (11:19 +0200)] 
README: build from source distro

Add url for source distro and build instructions.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 months agoCopyright NOTICE: change to use actual year
Jonas Pfefferle [Mon, 23 Apr 2018 09:11:41 +0000 (11:11 +0200)] 
Copyright NOTICE: change to use actual year

Change copyright notice for the Apache Software Foundation
to use the actual year instead of "onwards".

https://issues.apache.org/jira/browse/CRAIL-28

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoMove release date in HISTORY
Jonas Pfefferle [Wed, 18 Apr 2018 07:45:15 +0000 (09:45 +0200)] 
Move release date in HISTORY

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoAdd license header to md files
Jonas Pfefferle [Mon, 16 Apr 2018 15:37:16 +0000 (17:37 +0200)] 
Add license header to md files

Add license header to md files and remove exclude from rat.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoMove copyright notice to NOTICE file.
Jonas Pfefferle [Mon, 16 Apr 2018 15:31:10 +0000 (17:31 +0200)] 
Move copyright notice to NOTICE file.

Remove copyright notice from license headers and move it
to NOTICE file.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoFix LICENSE file
Jonas Pfefferle [Fri, 13 Apr 2018 08:56:04 +0000 (10:56 +0200)] 
Fix LICENSE file

LICENSE file should only include licenses that we use.
Now only includes Apache License 2.0

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoCheck licenses with apache-rat
Jonas Pfefferle [Fri, 13 Apr 2018 08:53:59 +0000 (10:53 +0200)] 
Check licenses with apache-rat

Add apache-rat maven plugin to allow to check licenses with
mvn apache-rat:check

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoAdd missing license headers
Jonas Pfefferle [Fri, 13 Apr 2018 08:52:48 +0000 (10:52 +0200)] 
Add missing license headers

https://issues.apache.org/jira/browse/CRAIL-25

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agov1.0 release history
Jonas Pfefferle [Thu, 12 Apr 2018 08:58:13 +0000 (10:58 +0200)] 
v1.0 release history

Close #19

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoREADME: update NVMf tier constants
Jonas Pfefferle [Tue, 10 Apr 2018 13:06:19 +0000 (15:06 +0200)] 
README: update NVMf tier constants

Update NVMf tier constants in README.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Close #18

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNew jNVMf version 1.3
Jonas Pfefferle [Tue, 10 Apr 2018 12:12:05 +0000 (14:12 +0200)] 
New jNVMf version 1.3

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNew DiSNI and DaRPC version
Jonas Pfefferle [Tue, 10 Apr 2018 12:03:32 +0000 (14:03 +0200)] 
New DiSNI and DaRPC version

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoCrail internal version to 3100
Jonas Pfefferle [Tue, 10 Apr 2018 11:08:59 +0000 (13:08 +0200)] 
Crail internal version to 3100

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfUnalignedWriteFuture: refactoring
Jonas Pfefferle [Tue, 10 Apr 2018 09:08:50 +0000 (11:08 +0200)] 
NvmfUnalignedWriteFuture: refactoring

Use offsetInSector in floorToSector and provide a new
floorToSector overload method for ints.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageEndpoint: Number of logical blocks fix type
Jonas Pfefferle [Tue, 10 Apr 2018 09:03:24 +0000 (11:03 +0200)] 
NvmfStorageEndpoint: Number of logical blocks fix type

jNVMf API has been changed to take int instead of long, fix code
accordingly.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStagingBufferCache: enhance excpetion message
Jonas Pfefferle [Tue, 10 Apr 2018 09:00:06 +0000 (11:00 +0200)] 
NvmfStagingBufferCache: enhance excpetion message

Add values of illegal arguments to exception message to make
it easier for the user to debug.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageConstants: Allocation size integer
Jonas Pfefferle [Tue, 10 Apr 2018 08:56:37 +0000 (10:56 +0200)] 
NvmfStorageConstants: Allocation size integer

Change allocationsize constant to be an integer. Since we do not
support allocation sizes > MAXINT.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageClient: isValid -> isAlive
Jonas Pfefferle [Tue, 10 Apr 2018 08:51:08 +0000 (10:51 +0200)] 
NvmfStorageClient: isValid -> isAlive

Change method name from isValid to isAlive. isValid can be
misinterpreted by the users that he misconfigured the system.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageClient: getEndpointGroup private
Jonas Pfefferle [Tue, 10 Apr 2018 08:48:23 +0000 (10:48 +0200)] 
NvmfStorageClient: getEndpointGroup private

Make getEndpointGroup private, it should not be used outside of
this class and does not handle concurrent execution.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmf Tier: keep alive timer constant
Jonas Pfefferle [Tue, 10 Apr 2018 08:46:03 +0000 (10:46 +0200)] 
Nvmf Tier: keep alive timer constant

Introduce new keep alive timer constant. For now we don't make
this configurable since jNVMf does not allow to change it yet, i.e.
it tells the target to use the default 2 minutes keep alive timer.
We set it to 110 seconds to not miss the 2 minutes deadline.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageClient: number of LBAs
Jonas Pfefferle [Mon, 9 Apr 2018 13:24:41 +0000 (15:24 +0200)] 
NvmfStorageClient: number of LBAs

Number of LBAs in new jNVMf API is no longer 0-based.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfRegisteredBufferCache: simplify
Jonas Pfefferle [Mon, 9 Apr 2018 13:22:41 +0000 (15:22 +0200)] 
NvmfRegisteredBufferCache: simplify

Simplify buffer cache by do not hold references to CrailBuffers.
We only return the corresponding key now instead of a wrapped
KeyedNativeBuffer.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageEndpoint: too many outstanding requests
Jonas Pfefferle [Mon, 9 Apr 2018 13:20:38 +0000 (15:20 +0200)] 
NvmfStorageEndpoint: too many outstanding requests

Fix bug were there could be too many outstanding requests posted
on the NVMf SQ.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfFuture: fix concurrency bug
Jonas Pfefferle [Mon, 9 Apr 2018 13:18:40 +0000 (15:18 +0200)] 
NvmfFuture: fix concurrency bug

Fix bug when completed counter was incremented in parallel
and never reached 2.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageClient: keep alive thread
Jonas Pfefferle [Mon, 9 Apr 2018 13:16:49 +0000 (15:16 +0200)] 
NvmfStorageClient: keep alive thread

Start keep alive thread which sends keep alive message to
each controller every 110s (timeout 120s). Otherwise thread sleeps.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStagingBufferCache: fix allocation
Jonas Pfefferle [Mon, 9 Apr 2018 13:13:57 +0000 (15:13 +0200)] 
NvmfStagingBufferCache: fix allocation

Fix allocation of Crail buffers to slice correct amount
of buffers for staging.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNvmfStorageTier: fix to work with new jNVMf API
Jonas Pfefferle [Wed, 4 Apr 2018 14:32:01 +0000 (16:32 +0200)] 
NvmfStorageTier: fix to work with new jNVMf API

Minor changes to work with latest jNVMf version.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoNew NVMf storage tier: use jNVMf library
Jonas Pfefferle [Thu, 1 Mar 2018 16:29:34 +0000 (17:29 +0100)] 
New NVMf storage tier: use jNVMf library

New NVMf storage tier implementation which uses jNVMf library instead
of SPDK. We do not implement unaligned reads anymore since we believe
the semantics of the underlying storage system should not be hidden like
this. We guarantee good performance when using buffered streams since
they now try to align accesses whenever possible.

https://issues.apache.org/jira/browse/CRAIL-22

Close #16

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoCrailStore: refactor alloc/free buffer interface
Jonas Pfefferle [Wed, 4 Apr 2018 08:52:46 +0000 (10:52 +0200)] 
CrailStore: refactor alloc/free buffer interface

Do not implement CrailBufferCache but instead use
abstract method to define allocate/free buffer interface.
Pass BufferCache instead of CrailStore to StorageClient init.

Close #14

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoBufferCache: use new interface in buffer cache
Jonas Pfefferle [Tue, 3 Apr 2018 13:50:59 +0000 (15:50 +0200)] 
BufferCache: use new interface in buffer cache

Use newly introduced buffer cache interface in buffer cache
implementation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoStorageClient: provide statistics and buffer cache
Jonas Pfefferle [Thu, 29 Mar 2018 12:27:37 +0000 (14:27 +0200)] 
StorageClient: provide statistics and buffer cache

Initialize the storage client with statistics and the buffer cache
to be used in the client.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-20

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoBufferedOutputStream: align CoreStream position
Jonas Pfefferle [Tue, 27 Mar 2018 11:53:37 +0000 (13:53 +0200)] 
BufferedOutputStream: align CoreStream position

After reopening an BufferedOutputStream the underlying CoreStream
might be at an arbitrary position since the stream is purged on close.
To avoid unnecessary operations on the underlying stream,
e.g. because of slices crossing the block boundary we adjust
the length of the first slice to align the position to slice size
such that after the first write all following writes on the underlying
stream will be aligned to slice size again. This also helps to avoid
unaligned writes in block storage tiers.

https://issues.apache.org/jira/browse/CRAIL-17

Close #11

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
5 months agoInclude test-jars and dependencies in assembly
Jonas Pfefferle [Wed, 28 Mar 2018 12:25:35 +0000 (14:25 +0200)] 
Include test-jars and dependencies in assembly

Add test-jars and test dependencies to jars directory.
Add test option to bin/crail to allow run unit tests from
command line.

https://issues.apache.org/jira/browse/CRAIL-18

Close #12

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
6 months agoConfiguration: environment variable expansion
Radu Stoica [Tue, 27 Feb 2018 14:09:45 +0000 (15:09 +0100)] 
Configuration: environment variable expansion

Enable environment variable expansion in crail-site.conf.

Close #7

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
6 months agoFixes issue with local short circuit (memcpy) path in the RDMA storage
Patrick Stuedi [Wed, 28 Feb 2018 08:41:47 +0000 (09:41 +0100)] 
Fixes issue with local short circuit (memcpy) path in the RDMA storage
tier
- mmaped buffers were previously indexed with wrong LBA in
  RdmaStorageLocalEndpoint

This fixes the CRAIL-3 JIRA ticket
https://issues.apache.org/jira/browse/CRAIL-3

Close #8

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
Signed-off-by: Adrian Schuepbach <asq@apache.org>
6 months agoApplications can now choose on a per node basis (files, directories,
Patrick Stuedi [Fri, 23 Feb 2018 14:44:49 +0000 (15:44 +0100)] 
Applications can now choose on a per node basis (files, directories,
tables, key/value pairs, etc.) whether the node should be enumerable or
not. If a node is enumerable it will be included in an enumeration
operation of the parent node.

https://issues.apache.org/jira/browse/CRAIL-10

Close #6

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
6 months agoNarpc: enable multiple core dispatcher
Patrick Stuedi [Fri, 23 Feb 2018 12:59:02 +0000 (13:59 +0100)] 
Narpc: enable multiple core dispatcher

Enable multi core dispatcher if configured to use NaRPC (RPC and
Storage).

Fixes CRAIL-9 JIRA ticket

Close #5

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
6 months agoDataNodeInfo.key() was not updated when the object got updated, this is
Patrick Stuedi [Thu, 22 Feb 2018 15:59:08 +0000 (16:59 +0100)] 
DataNodeInfo.key() was not updated when the object got updated, this is
fixed now.

https://issues.apache.org/jira/browse/CRAIL-8

Close #4

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
6 months ago[BUILD] Moving the default build to hadoop 2.7.3
Animesh Trivedi [Wed, 21 Feb 2018 12:56:00 +0000 (13:56 +0100)] 
[BUILD] Moving the default build to hadoop 2.7.3

Close #3

Signed-off-by: Animesh Trivedi <atrivedi@apache.org>
6 months agoUpdating README.md, how to contribute to Crail
Animesh Trivedi [Wed, 21 Feb 2018 12:46:27 +0000 (13:46 +0100)] 
Updating README.md, how to contribute to Crail

The website is also updated.

This closes #2

Signed-off-by: Animesh Trivedi <atrivedi@apache.org>
6 months agoFixing CRAIL-4 JIRA ticket
Animesh Trivedi [Wed, 7 Feb 2018 12:34:00 +0000 (13:34 +0100)] 
Fixing CRAIL-4 JIRA ticket

readShort and readLong functions by did not return
the expected types. Now that is fixed in the function
signature.

This closes #1

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
7 months agoAdjusting README (github links, source code links)
Patrick Stuedi [Mon, 22 Jan 2018 13:59:46 +0000 (14:59 +0100)] 
Adjusting README (github links, source code links)

7 months agoFixes bug when using the TCP RPC binding together with multiple namenodes
Patrick Stuedi [Fri, 19 Jan 2018 12:06:05 +0000 (13:06 +0100)] 
Fixes bug when using the TCP RPC binding together with multiple namenodes

8 months agoMoving to DaRPC and DiSNI v1.4
Patrick Stuedi [Thu, 18 Jan 2018 15:35:00 +0000 (16:35 +0100)] 
Moving to DaRPC and DiSNI v1.4

8 months agoSupport configuration with cachelimit=0
Patrick Stuedi [Thu, 18 Jan 2018 13:40:03 +0000 (14:40 +0100)] 
Support configuration with cachelimit=0

8 months agoAdjusting license and configuration (Apache)
Patrick Stuedi [Thu, 18 Jan 2018 13:27:40 +0000 (14:27 +0100)] 
Adjusting license and configuration (Apache)

8 months agoRenaming artifacts (Apache)
Patrick Stuedi [Thu, 18 Jan 2018 12:01:50 +0000 (13:01 +0100)] 
Renaming artifacts (Apache)

8 months agoApache package renaming
Patrick Stuedi [Thu, 18 Jan 2018 11:41:13 +0000 (12:41 +0100)] 
Apache package renaming

8 months agoTCP based RPC and storage backend
Patrick Stuedi [Mon, 8 Jan 2018 10:45:06 +0000 (11:45 +0100)] 
TCP based RPC and storage backend

8 months agoFixing bug with invalid flag not passed when writing directory records
Patrick Stuedi [Wed, 20 Dec 2017 15:13:11 +0000 (16:13 +0100)] 
Fixing bug with invalid flag not passed when writing directory records

9 months agotiny fix
follitude [Mon, 18 Dec 2017 09:59:57 +0000 (17:59 +0800)] 
tiny fix

9 months agoRemoving old key/value interface
Patrick Stuedi [Tue, 5 Dec 2017 13:04:27 +0000 (14:04 +0100)] 
Removing old key/value interface

9 months agoNvmf storage tier: target independence
Jonas Pfefferle [Fri, 24 Nov 2017 15:05:03 +0000 (16:05 +0100)] 
Nvmf storage tier: target independence

Make code NVMf target independent, i.e. the storage tier
can now be run with any target, e.g. SPDK, kernel, etc.
The storage tier connects to a running target to get
volume information and registers it to the namenode.
Before the code would actually start a NVMf target which
one now has to do by himself.

9 months agoSupport for key/value files and tables
Patrick Stuedi [Fri, 10 Nov 2017 13:44:48 +0000 (14:44 +0100)] 
Support for key/value files and tables

10 months agoSupport for persisting namenode
Patrick Stuedi [Fri, 27 Oct 2017 14:39:08 +0000 (16:39 +0200)] 
Support for persisting namenode

10 months agoSupporting restart of datanodes
Patrick Stuedi [Wed, 25 Oct 2017 12:30:11 +0000 (14:30 +0200)] 
Supporting restart of datanodes

10 months agoAdding logical block address information
Patrick Stuedi [Tue, 24 Oct 2017 14:41:39 +0000 (16:41 +0200)] 
Adding logical block address information

10 months agoUpdate README.md
Patrick Stuedi [Fri, 27 Oct 2017 08:19:52 +0000 (10:19 +0200)] 
Update README.md

10 months agoMerge pull request #22 from animeshtrivedi/master
Patrick Stuedi [Wed, 25 Oct 2017 11:24:51 +0000 (13:24 +0200)] 
Merge pull request #22 from animeshtrivedi/master

Adding missing get/put Float function on buffered streams

12 months agomoving to Disni 1.3 and DaRPC 1.3
Patrick Stuedi [Tue, 19 Sep 2017 08:17:16 +0000 (10:17 +0200)] 
moving to Disni 1.3 and DaRPC 1.3

12 months agoMake sure storage properties are set when initializing the NVMf storage server
Patrick Stuedi [Fri, 15 Sep 2017 12:36:12 +0000 (14:36 +0200)] 
Make sure storage properties are set when initializing the NVMf storage server

12 months agoFix bug in random read benchmark when switching between buffered and direct streams
Patrick Stuedi [Wed, 13 Sep 2017 15:27:08 +0000 (17:27 +0200)] 
Fix bug in random read benchmark when switching between buffered and direct streams

12 months agowrong returned data type for float
Animesh Trivedi [Wed, 13 Sep 2017 08:15:18 +0000 (10:15 +0200)] 
wrong returned data type for float

12 months agoMerge branch 'master' of github.com:zrlio/crail
Animesh Trivedi [Fri, 8 Sep 2017 09:31:50 +0000 (11:31 +0200)] 
Merge branch 'master' of github.com:zrlio/crail

12 months agoKeyValue client benchmark
Patrick Stuedi [Fri, 8 Sep 2017 09:18:47 +0000 (11:18 +0200)] 
KeyValue client benchmark

12 months agoCleaning up Storage and RPC interfaces
Patrick Stuedi [Wed, 6 Sep 2017 15:27:25 +0000 (17:27 +0200)] 
Cleaning up Storage and RPC interfaces

12 months ago(a) Fixing hash function when dispatching multiple namenodes
Patrick Stuedi [Wed, 6 Sep 2017 15:19:31 +0000 (17:19 +0200)] 
(a) Fixing hash function when dispatching multiple namenodes
(b) Separating server and client interfaces for RPC and storage