zookeeper.git
2 days agoZOOKEEPER-3068: Improve C client logging of IPv6 hosts master
Brian Nixon [Fri, 20 Jul 2018 15:40:42 +0000 (17:40 +0200)] 
ZOOKEEPER-3068: Improve C client logging of IPv6 hosts

Author: Brian Nixon <nixon@fb.com>
Author: enixon <brian.nixon.cs@gmail.com>

Reviewers: andor@apache.org

Closes #547 from enixon/ZOOKEEPER-3068 and squashes the following commits:

63eb18ef [enixon] Merge branch 'master' into ZOOKEEPER-3068
e3cf7f83 [Brian Nixon] change boolean from int to char
cd404534 [Brian Nixon] ZOOKEEPER-3068: Improve C client logging of IPv6 hosts

3 days agoZOOKEEPER-3093: sync zerror with ZOO_ERRORS
Kent R. Spillner [Thu, 19 Jul 2018 17:28:38 +0000 (10:28 -0700)] 
ZOOKEEPER-3093: sync zerror with ZOO_ERRORS

I noticed that zerror(ZNOWATCHER) returns "unknown error" and when I went to investigate why I realized that actually three new values were added to ZOO_ERRORS enum without corresponding cases in zerror.  While here, also take the opportunity to shuffle the ordering of cases in zerror to match ZOO_ERRORS in order to make it easier to visually compare the two in the future to see if any values are missing cases (e.g. the fact that ZNEWCONFIGNOQUORUM and ZRECONFIGINPROGRESS were added to the bottom of the switch made it harder to see which all cases were missing).

Author: Kent R. Spillner <kspillner@acm.org>

Reviewers: phunt@apache.org

Closes #575 from sl4mmy/zookeeper-3093

Change-Id: I52eaaad1487aa25c2ffd101d4e8f7a365e59a3ae

3 days agoZOOKEEPER-3077: build outside of source directory
Kent R. Spillner [Thu, 19 Jul 2018 17:14:59 +0000 (10:14 -0700)] 
ZOOKEEPER-3077: build outside of source directory

Allow building native C library outside of source directory by explicitly
looking for jute generated header and source files inside source directory.

Signed-off-by: Kent R. Spillner <kspillneracm.org>
Author: Kent R. Spillner <kspillner@acm.org>

Reviewers: phunt@apache.org

Closes #557 from sl4mmy/zookeeper-3077

Change-Id: I7dafcaed6367362f3d8df283e7d0ef5804774ddc

3 days agoZOOKEEPER-3046: increased test timeout
Bogdan Kanivets [Thu, 19 Jul 2018 16:58:12 +0000 (09:58 -0700)] 
ZOOKEEPER-3046: increased test timeout

- bump up timeout to avoid too many build failures

Author: Bogdan Kanivets <bkanivets@gmail.com>

Reviewers: phunt@apache.org

Closes #576 from lavacat/test-timeouts-3.5

Change-Id: If0b4b45c33e1909120ad15454f1c2f3f242237b6
(cherry picked from commit b78e83b1d7101b9fe76dd0b1ed8e7af37f8a587e)
Signed-off-by: Patrick Hunt <phunt@apache.org>
3 days agoZOOKEEPER-3083: Remove some redundant and noisy log lines
Brian Nixon [Thu, 19 Jul 2018 03:59:14 +0000 (20:59 -0700)] 
ZOOKEEPER-3083: Remove some redundant and noisy log lines

Author: Brian Nixon <nixon@fb.com>

Reviewers: Benjamin Reed <breed@apache.org>, Michael Han <hanm@apache.org>, Andor Molnar <andor@cloudera.com>, 毛蛤丝 <maoling199210191@sina.com>

Closes #561 from enixon/ZOOKEEPER-3083

4 days agoZOOKEEPER-3074: Flaky test:org.apache.zookeeper.server.ServerStatsTest.testLatencyMetrics
maoling [Wed, 18 Jul 2018 13:49:21 +0000 (15:49 +0200)] 
ZOOKEEPER-3074: Flaky test:org.apache.zookeeper.server.ServerStatsTest.testLatencyMetrics

more details in [ZOOKEEPER-3074](https://issues.apache.org/jira/browse/ZOOKEEPER-3074)

Author: maoling <maoling199210191@sina.com>

Reviewers: hanm@apache.org, andor@apache.org, nkalmar@cloudera.com

Closes #556 from maoling/ZOOKEEPER-3074

5 days agoZOOKEEPER-3042: testFailedTxnAsPartOfQuorumLoss is flaky
Bogdan Kanivets [Tue, 17 Jul 2018 08:41:24 +0000 (10:41 +0200)] 
ZOOKEEPER-3042: testFailedTxnAsPartOfQuorumLoss is flaky

- relaxed check of outstanding proposals queue
- close clients after restart
- restart client after old leader restart

Author: Bogdan Kanivets <bkanivets@gmail.com>

Reviewers: andor@apache.org

Closes #571 from lavacat/testFailedTxnAsPartOfQuorumLoss-fix-master

6 days agoZOOKEEPER-2184: Zookeeper Client should re-resolve hosts when connection attempts...
Andor Molnar [Mon, 16 Jul 2018 03:45:45 +0000 (20:45 -0700)] 
ZOOKEEPER-2184: Zookeeper Client should re-resolve hosts when connection attempts fail

This is the master/3.5 port of #451

Author: Andor Molnar <andor@cloudera.com>
Author: Andor Molnar <andor@apache.org>

Reviewers: Michael Han <hanm@apache.org>, Flavio Junqueira <fpj@apache.org>, Edward Ribeiro <edward.ribeiro@gmail.com>, Mark Fenes <mfenes@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #534 from anmolnar/ZOOKEEPER-2184_master

9 days agoZOOKEEPER-3033: MAVEN MIGRATION - Step 1.2 - create zk-recipes maven structure
Norbert Kalmar [Fri, 13 Jul 2018 12:58:03 +0000 (14:58 +0200)] 
ZOOKEEPER-3033: MAVEN MIGRATION - Step 1.2 - create zk-recipes maven structure

Create the zk-recipes maven structure for the maven migration.
Tested the recipes build with the new directory structure.

Author: Norbert Kalmar <nkalmar@yahoo.com>

Reviewers: andor@apache.org

Closes #564 from nkalmar/ZOOKEEPER-3033 and squashes the following commits:

c8c2e986 [Norbert Kalmar] ZOOKEEPER-3033 rename sub directories as well
d2b7f18d [Norbert Kalmar] ZOOKEEPER-3033 create zk-recipes maven structure

9 days agoZOOKEEPER-3087: Fix findbug warning introduced by ZOOKEEPER-3084.
Michael Han [Fri, 13 Jul 2018 10:47:39 +0000 (12:47 +0200)] 
ZOOKEEPER-3087: Fix findbug warning introduced by ZOOKEEPER-3084.

Author: Michael Han <hanm@apache.org>

Reviewers: andor@apache.org

Closes #568 from hanm/ZOOKEEPER-3087

10 days agoZOOKEEPER-2940: Deal with maxbuffer as it relates to large requests from clients
Andor Molnar [Wed, 11 Jul 2018 23:33:51 +0000 (16:33 -0700)] 
ZOOKEEPER-2940: Deal with maxbuffer as it relates to large requests from clients

https://issues.apache.org/jira/browse/ZOOKEEPER-2940

This PR covers the other part of Jute buffer monitoring which relates to client response sizes.

`jute.maxbuffer` is often set too small or the default (4MB) value is not enough to receive large responses from the server which causes the client unable to operate. e.g. `getChildren()` on a node which has lots of children or execution of a large multi.

These new metrics lets operators to monitor the size of generated responses to get an idea on how to properly set client's `jute.maxbuffer`.

The other part of monitoring which relates to proposal size has already been merged https://github.com/apache/zookeeper/pull/415

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #466 from anmolnar/ZOOKEEPER-2940 and squashes the following commits:

29224d021 [Andor Molnar] ZOOKEEPER-2940. Test fixes
486d4961d [Andor Molnar] ZOOKEEPER-2940. Removed support of 4lw commands, because master is targeted
60916e09e [Andor Molnar] ZOOKEEPER-2940. Code review fixes
b39c4c815 [Andor Molnar] ZOOKEEPER-2940. Code review: variable names renamed to proposalStats
87e72ff31 [Andor Molnar] ZOOKEEPER-2940. Some code review cleanups
6f0adb124 [Andor Molnar] ZOOKEEPER-2940. Code review fixes: stats class renamed, added comments to unit test asserts, updated docs
bcb6d4ca5 [Andor Molnar] ZOOKEEPER-2940. Added new metrics to python files
77a5f0aa8 [Andor Molnar] ZOOKEEPER-2940. Fixed failing unit tests
2def61c30 [Andor Molnar] ZOOKEEPER-2940. Track size of client responses

Change-Id: I68df371097cc355b1f41b8dbb0168da4fa5dee43

11 days agoZOOKEEPER-3084: Exit when ZooKeeper cannot bind to the leader election port
Fangmin Lyu [Wed, 11 Jul 2018 04:04:17 +0000 (21:04 -0700)] 
ZOOKEEPER-3084: Exit when ZooKeeper cannot bind to the leader election port

Author: Fangmin Lyu <allenlyu@fb.com>

Reviewers: Andor Molnár <andor@apache.org>, Benjamin Reed <breed@apache.org>, Norbert Kalmar <nkalmar@cloudera.com>, Michael Han <hanm@apache.org>

Closes #562 from lvfangmin/ZOOKEEPER-3084

12 days agoZOOKEEPER-3073: fix couple of typos
Christine Poerschke [Tue, 10 Jul 2018 11:29:20 +0000 (04:29 -0700)] 
ZOOKEEPER-3073: fix couple of typos

Author: Christine Poerschke <cpoerschke@bloomberg.net>

Reviewers: Norbert Kalmar <nkalmar@cloudera.com>, 毛蛤丝 <maoling199210191@sina.com>, Benjamin Reed <breed@apache.org>

Closes #554 from cpoerschke/master-ZOOKEEPER-3073

12 days agoZOOKEEPER-3078: remove print_completion_queue
Kent R. Spillner [Tue, 10 Jul 2018 10:58:32 +0000 (03:58 -0700)] 
ZOOKEEPER-3078: remove print_completion_queue

The fprintf(LOGSTREAM, ...) calls cause compiler errors when building
with GCC 8: "error: null argument where non-null required".  This is
because since ZOOKEEPER-1400 (2013-05-19 commit 37973fab: Allow logging
via callback instead of raw FILE pointer) LOGSTREAM is defined as NULL.
However, the last use of print_completion_queue was removed 2008-01-10
in commit ffc4b0d4 (Changed zoo_get() signature, changed the watcher
callback signature, fixed zookeeper_close() resource leaks and race
conditions, fixed the race condition causing xid mismatch).  So we can
avoid the compiler errors by just removing print_completion_queue.

Signed-off-by: Kent R. Spillner <kspillneracm.org>
Author: Kent R. Spillner <kspillner@acm.org>

Reviewers: Norbert Kalmar <nkalmar@cloudera.com>, Andor Molnár <andor@apache.org>, Benjamin Reed <breed@apache.org>

Closes #558 from sl4mmy/zookeeper-3078

12 days agoZOOKEEPER-3079: avoid unsafe use of sprintf(3)
Kent R. Spillner [Tue, 10 Jul 2018 10:50:07 +0000 (03:50 -0700)] 
ZOOKEEPER-3079: avoid unsafe use of sprintf(3)

The function format_endpoint_info declares both addrstr and buf as 128
element char arrays, however on non-Windows platforms it calls
sprintf(3) to write into buf the value of addrstr followed by ':'
followed by the the port number.  This causes a compiler error when
building with GCC 8 because this could potentially overflow buf if the
value of addrstr was ever 127 characters long (or a little less
depending on how many digits are in port).  Of course, this couldn't
actually happen because addrstr is initialized by inet_ntop(3) which
won't write more than INET6_ADDRSTRLEN bytes (defined in <netinet/in.h>
on POSIX-compliant systems).  Of course, GCC doesn't know that, so let's
just declare addrstr as a char array of only size INET6_ADDRSTRLEN
instead of 128.

Signed-off-by: Kent R. Spillner <kspillneracm.org>
Author: Kent R. Spillner <kspillner@acm.org>

Reviewers: Benjamin Reed <breed@apache.org>

Closes #559 from sl4mmy/zookeeper-3079

12 days agoZOOKEEPER-2886: Permanent session moved error in multi-op only conne…
Fangmin Lyu [Tue, 10 Jul 2018 10:46:36 +0000 (03:46 -0700)] 
ZOOKEEPER-2886: Permanent session moved error in multi-op only conne…

…ctions

Author: Fangmin Lyu <allenlyu@fb.com>

Reviewers: Andor Molnár <andor@apache.org>, Benjamin Reed <breed@apache.org>

Closes #353 from lvfangmin/ZOOKEEPER-2886

2 weeks agoZOOKEEPER-2873: abort startup on invalid ports
Norbert Kalmar [Wed, 4 Jul 2018 15:11:39 +0000 (17:11 +0200)] 
ZOOKEEPER-2873: abort startup on invalid ports

This change will check each server.x config if the client and election port are the same. Currently, ZK will startup and a race for the ports will take place. There will be an error, but ZK will keep running (without the ability to elect a leader). Now, ZK will fail on startup.

Author: Norbert Kalmar <nkalmar@yahoo.com>

Reviewers: Andor Molnar <andor@apache.org>

Closes #549 from nkalmar/ZOOKEEPER-2873 and squashes the following commits:

022c8b70 [Norbert Kalmar] ZOOKEEPER-2873 modify test case to expect exception
60b8128d [Norbert Kalmar] ZOOKEEPER-2873 fix old test - ReconfigTest.testUnspecifiedClientAddress
3d747c08 [Norbert Kalmar] ZOOKEEPER-2873 abort startup on invalid ports

2 weeks agoZOOKEEPER-3022: MAVEN MIGRATION - Iteration 1 - docs, it
Norbert Kalmar [Wed, 4 Jul 2018 11:02:19 +0000 (13:02 +0200)] 
ZOOKEEPER-3022: MAVEN MIGRATION - Iteration 1 - docs, it

First iteration of the directory changes in maven migration.
This time, the empty zookeeper-it for future integration tests were created, and the docs source were moved to zookeeper-docs.
Tested with forest v0.9, documents generations were successful with ant.

Author: Norbert Kalmar <nkalmar@yahoo.com>

Reviewers: Andor Molnar <andor@apache.org>

Closes #550 from nkalmar/ZOOKEEPER-3022-1

3 weeks agoZOOKEEPER-3066: Expose on JMX of Followers the id of the current leader
Enrico Olivelli [Tue, 26 Jun 2018 09:18:38 +0000 (11:18 +0200)] 
ZOOKEEPER-3066: Expose on JMX of Followers the id of the current leader

Expose a new JMX attribute "isLeader" in RemotePeerBean and LocalPeerBean

Author: Enrico Olivelli <eolivelli@apache.org>

Reviewers: Andor Molnar <andor@apache.org>

Closes #546 from eolivelli/fix/ZOOKEEPER-3066-leader-jmx and squashes the following commits:

bf09e37d [Enrico Olivelli] clean up spaces
ded4ce20 [Enrico Olivelli] reorder imports
81d7bb1c [Enrico Olivelli] split tests into LocalPeerBeanTest and QuorumPeerTest
a1984ef1 [Enrico Olivelli] fix typo
eb8d1bea [Enrico Olivelli] split tests
86b116b0 [Enrico Olivelli] enhance tests
b5000efc [Enrico Olivelli] clean up test
43782fe5 [Enrico Olivelli] introduce mockito tests
2778e742 [Enrico Olivelli] revert "ZOOKEEPER-3066 add servers restart to test case"
5f120631 [Enrico Olivelli] ZOOKEEPER-3066 add servers restart to test case
067aed9e [Enrico Olivelli] ZOOKEEPER-3066 Expose on JMX of Followers the id of the current leader Enhance RemotePeerMXBean and LocalPeerMXBean by adding information about leadership

3 weeks agoZOOKEEPER-3059: EventThread leak in case of Sasl AuthFailed
Abhishek Singh Chouhan [Mon, 25 Jun 2018 10:30:19 +0000 (12:30 +0200)] 
ZOOKEEPER-3059: EventThread leak in case of Sasl AuthFailed

Since authFailed is similar to session expired and is considered a fatal event, we should clean up after ourselves once we get a AuthFailed, other wise this results in an unavoidable and un-cleanable thread leak  of EventThread since the close operation is also a no-op (we return after checking for isAlive).

Author: Abhishek Singh Chouhan <abhishekchouhan121@gmail.com>

Reviewers: Andor Molnar <andor@apache.org>

Closes #541 from abhishek-chouhan/master and squashes the following commits:

c54a83a4 [Abhishek Singh Chouhan] ZOOKEEPER-3059 EventThread leak in case of Sasl AuthFailed. Adding testcase for the scenario
c1d9d7af [Abhishek Singh Chouhan] ZOOKEEPER-3059 EventThread leak in case of Sasl AuthFailed

4 weeks ago[ZOOKEEPER-2368] Send a watch event is when a client is closed
Tim Ward [Wed, 20 Jun 2018 13:22:07 +0000 (15:22 +0200)] 
[ZOOKEEPER-2368] Send a watch event is when a client is closed

Currently, if the client is closed (rather than being remotely disconnected) there is no notification to the watcher. This means that asynchronous clients can end up waiting indefinitely for events that will never come. Watchers need to be aware that the client is closed for good.

Signed-off-by: Tim Ward <timothyjwardapache.org>
Note that this is a variation on a patch I produced some time ago, which was broadly accepted as a good idea, and didn't cause any problems for Curator, but was deemed by some to be too risky because it reused an existing KeeperState. This patch is therefore updated to use a new `Closed` KeeperState. Fixing this would allow me to avoid maintaining a separate fork of Zookeeper just to support this one feature!

Author: Tim Ward <timothyjward@apache.org>

Reviewers: Andor Molnar <andor@apache.org>

Closes #529 from timothyjward/ZOOKEEPER-2368 and squashes the following commits:

d7196d19 [Tim Ward] Review comments from @anmolnar
088056b4 [Tim Ward] Review comments from @enixon
7fad1d36 [Tim Ward] [ZOOKEEPER-2368] Send a watch event is when a client is closed

(cherry picked from commit 6748a0e3f58f2a398dec4c6988bc70ea4363b807)
Signed-off-by: Andor Molnar <andor@cloudera.com>
5 weeks agoZOOKEEPER-3009: Potential NPE in NIOServerCnxnFactory
LJ1043041006 [Fri, 15 Jun 2018 06:02:41 +0000 (23:02 -0700)] 
ZOOKEEPER-3009: Potential NPE in NIOServerCnxnFactory

We have developed a static analysis tool [NPEDetector](https://github.com/lujiefsi/NPEDetector) to find some potential NPE. Our analysis shows that NPE reason can be simple:some callees may return null directly in corner case(e.g. node crash , IO exception), some of their callers have  !=null check but some do not have.
### Bug:
Callee getSocketAddress can return null, may caused by node crash, network exception....
<pre>
    public InetAddress getSocketAddress() {
        if (sock.isOpen() == false) {
            return null;
        }
        return sock.socket().getInetAddress();
    }
</pre>
getSocketAddress has two callers:  removeCnxn and removeCnxn

caller removeCnxn has null check
<pre>
 public boolean removeCnxn(NIOServerCnxn cnxn) {
        //other code
        InetAddress addr = cnxn.getSocketAddress();
        if (addr != null) {
            Set<NIOServerCnxn> set = ipMap.get(addr);
            //other code
        }
     //other code
    }
</pre>

but caller addCnxn has the similar code, but don't have the null check:
<pre>
     private void addCnxn(NIOServerCnxn cnxn) {
         InetAddress addr = cnxn.getSocketAddress();
         Set<NIOServerCnxn> set = ipMap.get(addr);
        //other code
      }
</pre>
I commit a patch to fix it: adding an null checker in addCnxn, and throws a semantics IOException rather than the ugly NPE.  I think the IOException  is ok, because the caller of addCnxn has the handler code:
<pre>
  private void processAcceptedConnections() {
               //other code
               try {
                    addCnxn(cnxn);
               } catch (IOException e) {
                    // register, createConnection
                    cleanupSelectionKey(key);
                    fastCloseSock(accepted);
               }
             }
</pre>
Maybe i am wrong, please point out my error.

Author: LJ1043041006 <1239497420@qq.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Allan Lyu <fangmin@apache.org>, Michael Han <hanm@apache.org>

Closes #525 from lujiefsi/ZOOKEEPER-3009

5 weeks agoZOOKEEPER-3063: Track outstanding changes with ArrayDeque
Yisong Yue [Fri, 15 Jun 2018 05:41:07 +0000 (22:41 -0700)] 
ZOOKEEPER-3063: Track outstanding changes with ArrayDeque

Changed outstandingChanges from ArrayList to ArrayDeque.

Author: Yisong Yue <yisongyue@fb.com>

Reviewers: Michael Han <hanm@apache.org>

Closes #543 from yisong-yue/ZOOKEEPER-3063

6 weeks agoZOOKEEPER-3019: add metric for slow fsyncs count
Norbert Kalmar [Fri, 8 Jun 2018 14:52:58 +0000 (16:52 +0200)] 
ZOOKEEPER-3019: add metric for slow fsyncs count

Added metric to count fsyncWarningThresholdMS exceed in FileTxnLog.commit(). Both jmx bean and Command can be used to get the stat.

Also updated the python standalone client and documentation.

Tested the py script, i.e.:
src/contrib/monitoring/check_zookeeper.py -o cacti -s "localhost:2181" -k zk_num_alive_connections
localhost_2181:1

rc/contrib/monitoring/check_zookeeper.py -o cacti -s "localhost:2181" -k zk_fsync_threshold_exceed_count
localhost_2181:0

mntr command now returns:
zk_version 3.6.0-SNAPSHOT-13c9f899ff62be6a2eacb28e522d06703f1dab08, built on 04/04/2018 12:09 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 7
zk_packets_sent 6
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state standalone
zk_znode_count 5
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 44
zk_open_file_descriptor_count 61
zk_max_file_descriptor_count 10240
zk_fsync_threshold_exceed_count 0

Author: Norbert Kalmar <nkalmar@yahoo.com>

Reviewers: Andor Molnar <andor@apache.org>

Closes #501 from nkalmar/ZOOKEEPER-3019 and squashes the following commits:

067edf6a [Norbert Kalmar] ZOOKEEPER-3019 remove redundant log
7430dbce [Norbert Kalmar] ZOOKEEPER-3019 remove metric from 4ltr command as no longer supported
1916a418 [Norbert Kalmar] ZOOKEEPER-3019 add extra warning log
761450f9 [Norbert Kalmar] ZOOKEEPER-3019 remove fsync from py script stat check
3b22bb7b [Norbert Kalmar] ZOOKEEPER-3019 remove fsyncEsceed from stats
0be003a8 [Norbert Kalmar] ZOOKEEPER-3019 exclude serverStats IS chec from findbugs
b48d2958 [Norbert Kalmar] ZOOKEEPER-3019 add unit test and fix PR comments
fb4c7a31 [Norbert Kalmar] ZOOKEEPER-3019 add metric to track slow fsyncs count + update py script and docs

7 weeks agoZOOKEEPER-2968: Add C client code coverage tests
Mark Fenes [Thu, 31 May 2018 19:57:21 +0000 (12:57 -0700)] 
ZOOKEEPER-2968: Add C client code coverage tests

ZOOKEEPER-2968: Add C client code coverage tests

This PR adds a new ant target 'c_coverage_report' which generates coverage report for the ZK C client.
The report is generated to build/c_coverage/report in html format.
As a requirement, lcov has to be installed prior to running target 'c_coverage_report'.

An additional check was added to 'check-cppunit-makefile' to ensure that the Makefile gets deleted and regenerated without the coverage compiler flags when running targets without --enable-gcov.

Author: Mark Fenes <mfenes@cloudera.com>

Reviewers: phunt@apache.org

Closes #467 from mfenes/ZOOKEEPER-2968 and squashes the following commits:

d8757821a [Mark Fenes] ZOOKEEPER-2968: Add C client code coverage tests
3c9022622 [Mark Fenes] ZOOKEEPER-2968: Add C client code coverage tests

Change-Id: Id4bd05afbff2447c593427eff5f133c3d88048d3

7 weeks agoZOOKEEPER-2920: Upgrade OWASP Dependency Check to 3.2.1
Patrick Hunt [Thu, 31 May 2018 03:54:52 +0000 (20:54 -0700)] 
ZOOKEEPER-2920: Upgrade OWASP Dependency Check to 3.2.1

Updated and verified on all the active branches.

Change-Id: I750d7e576e8c731aab4cef26f6863eac6b1b8dc2

Author: Patrick Hunt <phunt@apache.org>

Reviewers: Michael Han <hanm@apache.org>

Closes #536 from phunt/ZOOKEEPER-2920

7 weeks agoZOOKEEPER-2317: Ensure that OSGi versions are valid
Tim Ward [Wed, 30 May 2018 00:23:11 +0000 (17:23 -0700)] 
ZOOKEEPER-2317: Ensure that OSGi versions are valid

Signed-off-by: Tim Ward <timothyjwardapache.org>
Author: Tim Ward <timothyjward@apache.org>

Reviewers: phunt@apache.org

Closes #531 from timothyjward/ZOOKEEPER-2317

Change-Id: I072db1e9de3bc1ad66e23be40bdea642866f6883
(cherry picked from commit 1c2c6849f0ebc6d56a1e987d3a637c0c050f7a53)
Signed-off-by: Patrick Hunt <phunt@apache.org>
7 weeks agoZOOKEEPER-3043: QuorumKerberosHostBasedAuthTest fails on Linux box: Unable to parse...
Enrico Olivelli [Wed, 30 May 2018 00:02:56 +0000 (17:02 -0700)] 
ZOOKEEPER-3043: QuorumKerberosHostBasedAuthTest fails on Linux box: Unable to parse:includedir /etc/krb5.conf.d/

Fixes QuorumKerberosHostBasedAuthTest, see ZOOKEEPER-3043

 - upgrade Kerby to 1.0.0
 - upgrade commons-io to 2.5, needed by Kerby
 - fix test dir in KerberosTestUtils

This should be also cherry-picked to 3.5 branch

Author: Enrico Olivelli <eolivelli@apache.org>

Reviewers: phunt@apache.org

Closes #524 from eolivelli/fix/ZOOKEEPER-3043 and squashes the following commits:

b4f31ad67 [Enrico Olivelli] upgrade libs
e224e823e [Enrico Olivelli] QuorumKerberosHostBasedAuthTest fails on Linux box: Unable to parse:includedir /etc/krb5.conf.d/  - upgrade Kerby to 1.0.0  - upgrade commons-io to 2.5, needed by Kerby  - fix test dir in KerberosTestUtils

Change-Id: Idaa0c05d1212c936b60fa2aa5ccb9dee114cce49

7 weeks agoZOOKEEPER-1919 Update the C implementation of removeWatches to have it match ZOOKEEPE...
Andor Molnar [Tue, 29 May 2018 23:27:43 +0000 (16:27 -0700)] 
ZOOKEEPER-1919 Update the C implementation of removeWatches to have it match ZOOKEEPER-1910

Existing patch on Jira has been rebased and applied to master.

Replaces #511

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #522 from anmolnar/ZOOKEEPER-1919 and squashes the following commits:

f81a7608a [Andor Molnar] ZOOKEEPER-1919. Fixed indentation
d6455b062 [Andor Molnar] ZOOKEEPER-1919. Test fixes
769677693 [Andor Molnar] ZOOKEEPER-1919. Changes in C code

Change-Id: I2597854e4b28196a5dc5dc89553ced41537663eb

8 weeks agoZOOKEEPER-2993: Removed 'generated' line from .gitignore
asutosh936 [Wed, 23 May 2018 03:42:47 +0000 (20:42 -0700)] 
ZOOKEEPER-2993: Removed 'generated' line from .gitignore

Author: asutosh936 <asutosh.pandya@hotmail.com>

Reviewers: Abraham Fine <afine@apache.org>, Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>

Closes #488 from asutosh936/ZOOKEEPER-2993

8 weeks agoZOOKEEPER-3051: Updated jackson to the latest version
Patrick Hunt [Wed, 23 May 2018 03:34:19 +0000 (20:34 -0700)] 
ZOOKEEPER-3051: Updated jackson to the latest version

Change-Id: Iec76a7df76f8e1928828fd716c4c25f48e887ba1

Author: Patrick Hunt <phunt@apache.org>

Reviewers: Michael Han <hanm@apache.org>

Closes #527 from phunt/ZOOKEEPER-3051

2 months agoZOOKEEPER-3050: update to the newest version of Jetty
Patrick Hunt [Tue, 22 May 2018 04:37:04 +0000 (21:37 -0700)] 
ZOOKEEPER-3050: update to the newest version of Jetty

Change-Id: I1f63c471e59a1ed9d1cf58e721fedf47452acc6b

Author: Patrick Hunt <phunt@apache.org>

Reviewers: Michael Han <hanm@apache.org>

Closes #526 from phunt/ZOOKEEPER-3050

2 months agoZOOKEEPER-3046: added junit timeout to precede 'ant' timeout
Bogdan Kanivets [Mon, 21 May 2018 18:11:40 +0000 (11:11 -0700)] 
ZOOKEEPER-3046: added junit timeout to precede 'ant' timeout

When 'ant' timeout triggers, output of the test isn't captured.
Related issue ZOOKEEPER-2916

Author: Bogdan Kanivets <bkanivets@gmail.com>

Reviewers: phunt@apache.org

Closes #523 from lavacat/test-timeouts

Change-Id: I4941451395bd4f35342582e9f4852d73104da138
(cherry picked from commit 831bfebaec2500c5672fd665020b517282d9a8a9)
Signed-off-by: Patrick Hunt <phunt@apache.org>
2 months agoZOOKEEPER-3041: Typo in error message, affects log analysis; charater --> character
Hugh O'Brien [Wed, 16 May 2018 17:30:31 +0000 (10:30 -0700)] 
ZOOKEEPER-3041: Typo in error message, affects log analysis; charater --> character

Author: Hugh O'Brien <hugh.obrien@jet.com>

Reviewers: phunt@apache.org

Closes #498 from hughobrienjet/master

Change-Id: I3f4145c5c3eb4052cd7f0859053bb3711dfdd64c

2 months agoZOOKEEPER-3039: TxnLogToolkit uses Scanner badly
Andor Molnar [Thu, 10 May 2018 20:14:24 +0000 (13:14 -0700)] 
ZOOKEEPER-3039: TxnLogToolkit uses Scanner badly

Fixed by creating a single Scanner for all queries in the main() method.

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #517 from anmolnar/ZOOKEEPER-3039 and squashes the following commits:

a35e2e280 [Andor Molnar] ZOOKEEPER-3039. Optimize imports
a196443cb [Andor Molnar] ZOOKEEPER-3039. Use the same Scanner for all queries

Change-Id: Icf66888d4e6ad902615ed8ffde58a5a8fdd41237

2 months agoZOOKEEPER-3038: Cleanup some nitpicks in TTL implementation
Andor Molnar [Thu, 10 May 2018 04:13:16 +0000 (21:13 -0700)] 
ZOOKEEPER-3038: Cleanup some nitpicks in TTL implementation

A few nitpicks which needs to be cleaned up:

1. Rename OldEphemeralType --> EphemeralTypeEmulate353
2. Remove unused method: getTTL()
3. Remove unused import from QuorumPeer

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #516 from anmolnar/ZOOKEEPER-3038

Change-Id: Iec537706a35863d53daa593da2b01f6e523ef466

2 months agoZOOKEEPER-2959: ignore accepted epoch and LEADERINFO ack from observers
Alexander Shraer [Thu, 10 May 2018 04:00:15 +0000 (21:00 -0700)] 
ZOOKEEPER-2959: ignore accepted epoch and LEADERINFO ack from observers

2 months agoZOOKEEPER-3012: Fix unit test: testDataDirAndDataLogDir should not use hardcode test...
Andor Molnar [Wed, 9 May 2018 22:14:18 +0000 (15:14 -0700)] 
ZOOKEEPER-3012: Fix unit test: testDataDirAndDataLogDir should not use hardcode test folders

A little bit more than just fixing the hardcoded folder names. Because the original issue was only on the 3.4 branch, the unit test has also been added to that branch only.

In this PR I add the fixed version of test to the master branch. Should be committed to 3.5 as well and I'll create a separate PR for 3.4

https://issues.apache.org/jira/browse/ZOOKEEPER-3012

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #514 from anmolnar/ZOOKEEPER-3012

Change-Id: Iabba2630008158580a121a007a5c45b43a301315

2 months agoZOOKEEPER-2901: TTL Nodes don't work with Server IDs > 127
randgalt [Wed, 9 May 2018 21:57:41 +0000 (14:57 -0700)] 
ZOOKEEPER-2901: TTL Nodes don't work with Server IDs > 127

There was a major oversight when TTL nodes were implemented. The session ID generator for each server is seeded with the configured Server ID in the high byte. TTL Nodes were using the highest bit to denote a TTL node when used in the ephemeral owner. This meant that Server IDs > 127 that created ephemeral nodes would have those nodes always considered TTL nodes (with the TTL being essentially a random number).

This PR fixes the issue by disabling TTL Nodes by default. They must now be enabled in zoo.cfg. When TTL Nodes are enabled, the max Server ID changes from 255 to 254. This allows the high byte of a session ID stored in the ephemeral owner to use 0xFF to denote a TTL node.

About this change:

- The doc has been updated to note that TTL nodes are disabled by default and must be enabled via config. Also, the docs explains that when TTL nodes are enabled the max Server ID becomes 254
- The TTL implementation has been updated to use 0xFF in the high byte of the ephemeralOwner to denote a TTL node. This decreases the max TTL by an insignificant amount
- PrepRequestProcessor now throws `KeeperException.UnimplementedException` when an attempt to create a TTL node is made but the server has not been configured to enable TTL Nodes.
- QuorumPeer throws a `RuntimeException` if TTL Nodes are enabled but the Server ID > 254
- Tests have been added to validate all of this

IMPORTANT NOTE: TTL Nodes created in 3.5.3 will revert to EPHEMERAL with this change. We need to discuss the impact of this and consider workarounds, etc.

Author: randgalt <jordan@jordanzimmerman.com>
Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #377 from Randgalt/ZOOKEEPER-2901

Change-Id: I6923a22ecf847710d4e7151daf8c044734873f6b

2 months agoZOOKEEPER-2982: Re-try DNS hostname -> IP resolution if node connection fails
Eron Wright [Tue, 8 May 2018 22:54:02 +0000 (00:54 +0200)] 
ZOOKEEPER-2982: Re-try DNS hostname -> IP resolution if node connection fails

This PR ports a fix from the 3.4 to the master branch, that was originally made in ZOOKEEPER-1506.

Closes ZOOKEEPER-2982.

Author: Eron Wright <eron.wright@gmail.com>

Reviewers: Andor Molnar <andor@cloudera.com>, Abraham Fine <afine@apache.org>, Flavio Junqueira <fpj@apache.org>

Closes #513 from EronWright/ZOOKEEPER-2982-master

2 months agoZOOKEEPER-2988: NPE triggered if server receives a vote for a server …
Brian Nixon [Mon, 30 Apr 2018 04:34:57 +0000 (21:34 -0700)] 
ZOOKEEPER-2988: NPE triggered if server receives a vote for a server …

…id not in their voting view

Author: Brian Nixon <nixon@fb.com>

Reviewers: Abraham Fine <afine@apache.org>, Michael Han <hanm@apache.org>, Edward Ribeiro <edward.ribeiro@gmail.com>

Closes #476 from enixon/ZOOKEEPER-2988

2 months agoZOOKEEPER-3027: Accidently removed public API of FileTxnLog.setPreallocSize()
Andor Molnar [Fri, 27 Apr 2018 18:28:15 +0000 (11:28 -0700)] 
ZOOKEEPER-3027: Accidently removed public API of FileTxnLog.setPreallocSize()

In my latest commit regarding TxnLogToolkit there's a refactor to outsource file padding logic from FileTxnLog to a separate class:

https://github.com/apache/zookeeper/commit/126fb0f22d701cad58bf3123bf7d8f2219e60387#diff-89717124564925d61d29dd817bcdd915L147

Unfortunately public static method setPreallocSize(int) has also been moved to the new class, but it's being actively used by hadoop-common project too:

https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java#L384

I'd like to submit a patch to revert the deleted method which is going to call the new one, but will keep backward compatibility with Hadoop.

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #509 from anmolnar/ZOOKEEPER-3027

Change-Id: I7333b5e24c2a78d10a5c5a74c181633050bd225d

2 months agoZOOKEEPER-3007: Potential NPE in ReferenceCountedACLCache#deserialize
gongleigl.gong [Thu, 26 Apr 2018 22:21:06 +0000 (15:21 -0700)] 
ZOOKEEPER-3007: Potential NPE in ReferenceCountedACLCache#deserialize

LJ1043041006 found a potential NPE in ZK

Author: gongleigl.gong <gongleigl.gong@qunar.com>
Author: brettkk <1099446733@qq.com>

Reviewers: phunt@apache.org

Closes #495 from brettKK/master and squashes the following commits:

7eb9e1c9b [gongleigl.gong] fix RTE message in ReferenceCountedACLCache class
0b858826d [gongleigl.gong] del logger error and fix error message
a12b13ff7 [brettkk] fix format
f7da9b9fa [brettkk] change ZOOK3007 to compare with apache master
cf9fb5f9a [brettkk] recover code
c4db5e25d [brettkk] recover zookeeper master same with apache:master
700dfb7f4 [gongleigl.gong] fix NPE bug
7d8d5230c [gongleigl.gong] d

Change-Id: I012c242e8566fcd54c1ebde1a30ec785b6aa31b0

2 months agoZOOKEEPER-2955: Enable Clover code coverage report
Mark Fenes [Wed, 25 Apr 2018 22:17:09 +0000 (15:17 -0700)] 
ZOOKEEPER-2955: Enable Clover code coverage report

ZOOKEEPER-2955: Enable Clover code coverage report

This PR configures OpenClover to generate Java code coverage reports.

To generate Java code coverage report run:
ant test-coverage-clover-java

For quick testing of this PR run:
ant -Dtestcase=test_file_name test-coverage-clover-java

Clover can also be run step-by-step:
ant -Drun.clover=true test-core-java
ant clover-report

Note: run.clover must not be set when building ZK for production use.

The reports will be placed under the build/test/clover/reports directory in HTML and XML formats.

Author: Mark Fenes <mfenes@cloudera.com>

Reviewers: phunt@apache.org

Closes #443 from mfenes/ZOOKEEPER-2955 and squashes the following commits:

321fc0152 [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report
ca68181b2 [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report
4cd1a3170 [Mark Fenes] Trigger notification
09872a66b [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report
3e0f2543c [Mark Fenes] Trigger notification
0cdb44442 [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report
270c12efc [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report
f59dcf8ad [Mark Fenes] ZOOKEEPER-2955: Enable Clover code coverage report

Change-Id: I089ca39224fa6012a299a46e4a35333ecc87ef35

2 months agoZOOKEEPER-2415: SessionTest is using Thread deprecated API
Andor Molnar [Tue, 24 Apr 2018 23:41:54 +0000 (16:41 -0700)] 
ZOOKEEPER-2415: SessionTest is using Thread deprecated API

Refactored `SessionTest` class.

`testSessionTimeout()` has been extracted to a separate class and split into multiple test methods to validate the different behaviours in a more readable way.

Removed deprecated Thread API `suspend()` and reduced the server ticktime / session timeout for better performance.

Hopefully this will help on test's flakyness too.

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #497 from anmolnar/ZOOKEEPER-2415 and squashes the following commits:

5f9f76c67 [Andor Molnar] ZOOKEEPER-2415. Reverted even more changes, because we don't need custom tickTime either anymore
8ed14e511 [Andor Molnar] ZOOKEEPER-2415. Moved session expiration test from ZooKeeperTestableTest class because it's redundant and more reliable
cfd18b64b [Andor Molnar] ZOOKEEPER-2415. Reset changes which are unrelated to this patch
76520025c [Andor Molnar] ZOOKEEPER-2415. Revert changes related to DisconnectableZooKeeper, because we don't need it for the new tests
6b5d26d0d [Andor Molnar] ZOOKEEPER-2415. Use ClientBase base class for the refactored unit tests
6026c591b [Andor Molnar] ZOOKEEPER-2415. Refactor testSessionTimeout() to live in separate class and not to use deprecated API. Also improved performance.

Change-Id: I33906f43de52a06e95ba3f19f8087033cd5857e2

2 months agoZOOKEEPER-3025: cmake broken on windows - make `hashtable` search `include`
Andrew Schwartzmeyer [Tue, 24 Apr 2018 00:25:44 +0000 (17:25 -0700)] 
ZOOKEEPER-3025: cmake broken on windows - make `hashtable` search `include`

When ZOOKEEPER-2999 removed the directory side-effect of
`include_directories(include)`, and added it as a target-level include
to the `zookeeper` library, this broke the Windows build. It worked on
Linux (where the patch was tested) because `winconfig.h` and is not
included, but on Windows, the `hashtable` library included
`winconfig.h` (found in `include`) but no longer knew where to find
it. The fix is to add `include` to the `hashtable` library's list of
include directories.

Author: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

Reviewers: phunt@apache.org

Closes #505 from andschwa/ZOOKEEPER-3025

Change-Id: I657f8e234082068fb7adf56dad2d68cbb4bc1d31

2 months agoZOOKEEPER-2994: Tool required to recover log and snapshot entries with CRC errors
Andor Molnar [Mon, 23 Apr 2018 22:20:26 +0000 (15:20 -0700)] 
ZOOKEEPER-2994: Tool required to recover log and snapshot entries with CRC errors

https://issues.apache.org/jira/browse/ZOOKEEPER-2994

In the event  of ZooKeeper transaction log becomes corrupted and fail CRC checks (preventing startup) we should have a mechanism to get the cluster running again.

Previously we achieved this by loading the broken transaction log with a modified version of ZK with disabled CRC check and forced it to write new txn log files.

It has proven that once you end up with the corrupt txn log there is no way to recover except manually modifying the crc check. That's basically why the tool is needed.

It's called TxnLogToolkit, a new console application similar to LogFormatter and SnapshotFormatter, but it's intentionally separated to keep backward compatibility in the existing tools.

This PR contains TXN log tool only.

You probably also notice a refactoring to extract file padding logic from FileTxnLog to reuse in the new tool. Related code changes can be reviewed alone in a separate commit if preferred.

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #487 from anmolnar/ZOOKEEPER-2994 and squashes the following commits:

221760ccc [Andor Molnar] ZOOKEEPER-2994. Added documentation and startup scripts
a69d7297b [Andor Molnar] ZOOKEEPER-2994. Fix findbugs warning
0b95efefd [Andor Molnar] ZOOKEEPER-2994. Fix for unit test
15fa45c68 [Andor Molnar] ZOOKEEPER-2994. Added padding, tool renamed to TxnLogToolkit, interactive mode, etc.
6a1ad0ec4 [Andor Molnar] ZOOKEEPER-2994. Refactor FileTxnLog's padding logic to separate class for reusability
0d089ccdd [Andor Molnar] ZOOKEEPER-2994. Added new tool TxnLogTool for txn log file recovery

Change-Id: I7560362633a7bc919ae6d3ca7e3588e196a1919c

3 months agoZOOKEEPER-3002: upgrade to jdk 8
Norbert Kalmar [Tue, 17 Apr 2018 04:57:24 +0000 (21:57 -0700)] 
ZOOKEEPER-3002: upgrade to jdk 8

Set minimum requirement to java 1.8
The code successfully compiled with 8.0.161-oracle and 8.0.163-zulu.

Checking the generated class file it reports 1.8 as version:
javap -verbose build/classes/org/apache/zookeeper/server/ServerStats.class | grep version
minor version: 0
major version: 52

Author: Norbert Kalmar <nkalmar@yahoo.com>

Reviewers: Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>

Closes #502 from nkalmar/ZOOKEEPER-3002_java1.8

3 months agoZOOKEEPER-2999: CMake build should use target-level commands
proller [Mon, 16 Apr 2018 04:29:43 +0000 (21:29 -0700)] 
ZOOKEEPER-2999: CMake build should use target-level commands

CMake is using `include_directories`, which has global side effects,
instead of the more explicit `target_include_directories`, to include
directories per target (and with private or public scoping).

Furthermore, CMake should also use `CMAKE_CURRENT_SOURCE_DIR` over
`CMAKE_SOURCE_DIR` in order to allow inclusion in other projects via
`add_subdirectory()`, and we can reduce the minimally required CMake
version to 3.5 from 3.6.

Author: proller <proller@github.com>

Reviewers: Michael Han <hanm@apache.org>

Closes #486 from andschwa/ZOOKEEPER-2999

3 months agoZOOKEEPER-3017: Link libm in CMake on FreeBSD.
David Forsythe [Mon, 16 Apr 2018 04:22:25 +0000 (21:22 -0700)] 
ZOOKEEPER-3017: Link libm in CMake on FreeBSD.

andschwa

Per your suggestion here: https://reviews.apache.org/r/66384/

Author: David Forsythe <dforsyth@freebsd.org>

Reviewers: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>, Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>

Closes #499 from dforsyth/ZOOKEEPER-3017

3 months agoZOOKEEPER-3006: Potential NPE in ZKDatabase#calculateTxnLogSizeLimit
Edward Ribeiro [Fri, 6 Apr 2018 04:04:07 +0000 (21:04 -0700)] 
ZOOKEEPER-3006: Potential NPE in ZKDatabase#calculateTxnLogSizeLimit

LJ1043041006 found a potential NPE in ZKDatabase#calculateTxnLogSizeLimit: https://issues.apache.org/jira/browse/ZOOKEEPER-3006

```
//ZKDatabase
public long calculateTxnLogSizeLimit() {
    long snapSize = 0;
    try {
        snapSize = snapLog.findMostRecentSnapshot().length();
     } catch (IOException e) {
        LOG.error("Unable to get size of most recent snapshot");
    }
    return (long) (snapSize * snapshotSizeFactor);
}
```

 in FileTxnSnapLog#findMostRecentSnapshot(), it will return the result of  FileSnap#findMostRecentSnapshot:

```
// called by FileTxnSnapLog#findMostRecentSnapshot()
public File findMostRecentSnapshot() throws IOException {
        List<File> files = findNValidSnapshots(1);
        if (files.size() == 0) {
            return null;
        }
        return files.get(0);
}
```

So it will return null when the files sizes is 0, but ZKDatabase#calculateTxnLogSizeLimit has no null checker.

Author: Edward Ribeiro <edward.ribeiro@gmail.com>

Reviewers: Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>, maoling <maoling199210191@sina.com>

Closes #493 from edwardoliveira/zk-3006-potential-npe

3 months agoZOOKEEPER-2997: CMake should not force static CRT linking
Andrew Schwartzmeyer [Mon, 26 Mar 2018 02:19:01 +0000 (19:19 -0700)] 
ZOOKEEPER-2997: CMake should not force static CRT linking

By removing this code, CMake will use its own defaults for the CRT
flags (e.g., `/MDd` for debug configurations). With it removed, the
user can override this behavior by setting the `CMAKE_CXX_FLAGS`
manually when configuring ZooKeeper.

Author: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

Reviewers: Michael Han <hanm@apache.org>

Closes #485 from andschwa/ZOOKEEPER-2997

3 months agoZOOKEEPER-2998: CMake declares incorrect ZooKeeper version
Andrew Schwartzmeyer [Mon, 26 Mar 2018 02:16:35 +0000 (19:16 -0700)] 
ZOOKEEPER-2998: CMake declares incorrect ZooKeeper version

This was not updated for the current development branch; it should be
3.6.0.

Author: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

Reviewers: Michael Han <hanm@apache.org>

Closes #484 from andschwa/ZOOKEEPER-2998

3 months agoZOOKEEPER-3001: Incorrect log message when try to delete container node
sel-fish [Mon, 26 Mar 2018 01:49:14 +0000 (18:49 -0700)] 
ZOOKEEPER-3001: Incorrect log message when try to delete container node

Missing `String.format`.

Author: sel-fish <fqlgy@hotmail.com>

Reviewers: Edward Ribeiro <edward.ribeiro@gmail.com>, Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>, maoling <maoling199210191@sina.com>

Closes #492 from sel-fish/ZOOKEEPER-3001

4 months agoZOOKEEPER-2992: The eclipse build target fails due to protocol redirection: http...
Shawn Heisey [Mon, 5 Mar 2018 02:51:25 +0000 (18:51 -0800)] 
ZOOKEEPER-2992: The eclipse build target fails due to protocol redirection: http->https

ZOOKEEPER-2992 is the Jira issue.

Author: Shawn Heisey <elyograg@apache.org>

Reviewers: Edward Ribeiro <edward.ribeiro@gmail.com>, Andor Molnár <andor@cloudera.com>, Michael Han <hanm@apache.org>

Closes #479 from elyograg/master

4 months agoZOOKEEPER-2936: Removed duplicate code
asutosh936 [Fri, 2 Mar 2018 22:12:17 +0000 (14:12 -0800)] 
ZOOKEEPER-2936: Removed duplicate code

Author: asutosh936 <asutosh.pandya@hotmail.com>

Reviewers: Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #473 from asutosh936/master and squashes the following commits:

6a17613be [asutosh936] ZOOKEEPER-2936 - Deleted duplicate code
a02ef3db8 [asutosh936] ZOOKEEPER-2936 - Removed duplicate code

4 months agoZOOKEEPER-2845: Apply commit log when restarting server.
Robert Evans [Fri, 23 Feb 2018 22:49:00 +0000 (14:49 -0800)] 
ZOOKEEPER-2845: Apply commit log when restarting server.

I will be creating a patch/pull request for 3.4 and 3.5 too, but I wanted to get a pull request up for others to look at ASAP.

I have a version of this based off of #310 at https://github.com/revans2/zookeeper/tree/ZOOKEEPER-2845-orig-test-patch but the test itself is flaky.  Frequently leader election does not go as planned on the test and it ends up failing but not because it ended up in an inconsistent state.

I am happy to answer any questions anyone has about the patch.

Author: Robert Evans <evans@yahoo-inc.com>

Reviewers: Abraham Fine <afine@apache.org>, Mark Fenes <mfenes@cloudera.com>, Andor Molnár <andor@cloudera.com>, Kishor Patil <kpatil@yahoo-inc.com>

Closes #453 from revans2/ZOOKEEPER-2845-master and squashes the following commits:

28c074a26 [Robert Evans] Addressed review comments
583e34435 [Robert Evans] Using framework APIs for test
f26a21ad6 [Robert Evans] Addressed review comments
93168d716 [Robert Evans] Added in a modified version of the test
3d042f981 [Robert Evans] ZOOKEEPER-2845: Apply commit log when restarting server.

4 months agoZOOKEEPER-2978: fix potential null pointer exception when deleting node
achimbab [Tue, 20 Feb 2018 22:38:00 +0000 (14:38 -0800)] 
ZOOKEEPER-2978: fix potential null pointer exception when deleting node

At line 518, 'existWatches.remove(clientPath)' is null because watches for clientPath is already removed.

Author: 07c00h <07c00h@gmail.com>

Reviewers: Abraham Fine <afine@apache.org>

Closes #461 from achimbab/master

5 months agoZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup
Mark Fenes [Wed, 14 Feb 2018 20:23:30 +0000 (12:23 -0800)] 
ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup

ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup

This PR adds a check to protect ZK against configuring dataDir and dataLogDir opposingly.

When FileTxnSnapLog is created, it checks if transaction log directory contains snapshot files or vice versa, snapshot directory contains transaction log files. If so, the check throws LogdirContentCheckException or SnapdirContentCheckException, respectively, which translates to DatadirException at ZK startup in QuorumPeerMain and ZooKeeperServerMain.

If the two directories are the same, then no check is done.

For testing, I've added 4 new unit tests which cover the following cases:

1. transaction log and snapshot directories are different and they are used correctly (no Exception)
2. transaction log and snapshot directories are the same (in this case no check is done)
3. transaction log and snapshot directories are different and transaction log directory contains snapshot files (LogdirContentCheckException -> ZK quits)
4. transaction log and snapshot directories are different and snapshot directory contains transaction log files (SnapdirContentCheckException -> ZK quits)

Author: Mark Fenes <mfenes@cloudera.com>

Reviewers: Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #450 from mfenes/ZOOKEEPER-2967 and squashes the following commits:

3f2ce609a [Mark Fenes] Re-run checks
3dba7f14a [Mark Fenes] ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup
50e7150b3 [Mark Fenes] Trigger notification
5f8ada87f [Mark Fenes] ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup
36fc5bd46 [Mark Fenes] ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup
eb76730cb [Mark Fenes] ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup
81c026fe8 [Mark Fenes] ZOOKEEPER-2967: Add check to validate dataDir and dataLogDir parameters at startup

5 months agoZOOKEEPER-2939: Added last/min/max proposal size JMX beans
Andor Molnár [Wed, 7 Feb 2018 00:05:18 +0000 (16:05 -0800)] 
ZOOKEEPER-2939: Added last/min/max proposal size JMX beans

https://issues.apache.org/jira/browse/ZOOKEEPER-2939
- Refactor proposal serialization logic to a common place (SerializeUtils),
- Add JMX metric to monitor jute.maxbuffer setting
- Add JMX metric to monitor min/max/last size of generated proposals
- Add proposal stats info to 'stat' 4lw command
- Add proposal stats resetting to 'srst' 4lw command
- Unit tests

Author: Andor Molnár <andor@cloudera.com>
Author: Andor Molnar <andor@cloudera.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Abraham Fine <afine@apache.org>

Closes #415 from anmolnar/ZOOKEEPER-2933 and squashes the following commits:

cf6ea5ecb [Andor Molnar] ZOOKEEPER-2933. Added proposal size statistics to 'mntr' command
cb28df6d6 [Andor Molnár] ZOOKEEPER-2939. Fixed findbugs issue: newline in format string
8e6b65f9f [Andor Molnár] ZOOKEEPER-2939: Added last/min/max proposal size JMX beans

5 months agoZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to...
Benedict Jin [Thu, 1 Feb 2018 23:26:58 +0000 (15:26 -0800)] 
ZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

`FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

------
Example from jenkins [log](https://builds.apache.org/job/PreCommit-ZOOKEEPER-github-pr-build/830/console):
```java
2017-06-28 10:25:16,645 [myid:] - WARN  [SyncThread:0:FileTxnLog341] - fsync-ing the write ahead log (1945616 bytes) in SyncThread:0 took 2240ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
```

Author: asdf2014 <benedictjin2016@gmail.com>
Author: asdf2014 <1571805553@qq.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #296 from asdf2014/ZOOKEEPER-2824 and squashes the following commits:

7cf4b6848 [asdf2014] Improve latency log description
dd2268f2f [asdf2014] ZOOKEEPER-2824: `FileChannel#size` info should be added to `FileTxnLog#commit` to solve the confuse that reason is too large log or too busy disk I/O

5 months agoZOOKEEPER-2949: using hostname and port to create SSLEngine
Feng Shaobao [Tue, 30 Jan 2018 20:56:37 +0000 (12:56 -0800)] 
ZOOKEEPER-2949: using hostname and port to create SSLEngine

If the server has more than one host name, and serve each host name with different certificates. then the ssl client must provide the server name in the ssl Hello packet, to tell the server which certificate to use.
This is especially important when the client connect to a load balancer with different backend services.

https://en.wikipedia.org/wiki/Server_Name_Indication

Author: f00231050 <shaobao.feng@huawei.com>

Reviewers: Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #423 from abel-von/ZOOKEEPER-2949

5 months agoZOOKEEPER-1580: QuorumPeer.setRunning is not used
maoling [Tue, 30 Jan 2018 07:02:41 +0000 (23:02 -0800)] 
ZOOKEEPER-1580: QuorumPeer.setRunning is not used

- more details in [JIRA:ZOOKEEPER-1580](https://issues.apache.org/jira/browse/ZOOKEEPER-1580)
- I forget fetching the upstream codes, make a mistake in the origin [PR-446] (https://github.com/apache/zookeeper/pull/446) which includes some review history,so I close it and open a new one

Author: maoling <maoling199210191@sina.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Andor Molnár <andor@cloudera.com>, Abraham Fine <afine@apache.org>

Closes #452 from maoling/ZOOKEEPER-1580 and squashes the following commits:

29a5aba1e [maoling] remove the setter and return back to setting the running field directly
28de1e88f [maoling] ZOOKEEPER-1580:QuorumPeer.setRunning is not used

6 months agoZOOKEEPER-2964: Conf" command returns dataDir and dataLogDir opposingly
Qihong Xu [Fri, 19 Jan 2018 00:12:03 +0000 (16:12 -0800)] 
ZOOKEEPER-2964: Conf" command returns dataDir and dataLogDir opposingly

See [https://issues.apache.org/jira/browse/ZOOKEEPER-2964](url) for details.

The bug affects versions newer than 3.5. According to Andor Molnar‘s [review](https://issues.apache.org/jira/browse/ZOOKEEPER-2964?focusedCommentId=16330018&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16330018) this patch can be applied to master and branch-3.5 branches.

Thanks all for reviewing this issue.

Author: Qihong Xu <qihongxu@bu.edu>

Reviewers: phunt@apache.org

Closes #449 from qihongxu/ZOOKEEPER-2964

Change-Id: Ie7de251afd5cca64f4735d75d9c95a886ff75328

6 months agoZOOKEEPER-2961: Fix testElectionFraud Flakyness
Abraham Fine [Thu, 18 Jan 2018 23:52:43 +0000 (15:52 -0800)] 
ZOOKEEPER-2961: Fix testElectionFraud Flakyness

This test relies on hooking into our logging system and creates a new appender using a PatternLayout object shared with the CONSOLE appender. PatternLayout has some synchronization issues (https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html) so we should create a new instance of it.

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #437 from afine/ZOOKEEPER-2961_master

Change-Id: I1c81d37b94b12e0721a671deb2fc983773f88fd9

6 months agoZOOKEEPER-2249: CRC check failed when preAllocSize smaller than node data
Abraham Fine [Thu, 18 Jan 2018 23:43:11 +0000 (15:43 -0800)] 
ZOOKEEPER-2249: CRC check failed when preAllocSize smaller than node data

This bug is caused by an issue in our file padding logic. We calculate an incorrect position to add padding when appending to the transaction log, this often corrupts a transaction. When the log is read the CRC check will correctly fail.

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #436 from afine/ZOOKEEPER-2249 and squashes the following commits:

f3bc06c1 [Abraham Fine] Improve testing and respond to phunt's comment
44ad0575 [Abraham Fine] ZOOKEEPER-2249: CRC check failed when preAllocSize smaller than node data

Change-Id: Id8ba9ad730760cb78672127b8c0e02db60b4e87d

7 months agoZOOKEEPER-2893: very poor choice of logging if client fails to connect to server
Andor Molnar [Tue, 19 Dec 2017 18:27:25 +0000 (10:27 -0800)] 
ZOOKEEPER-2893: very poor choice of logging if client fails to connect to server

'addr' variable is used to identify which server to connect to.
I've made this available for error handling code in order to let it fallback to this address if the remote socket hasn't been initialised yet. This will give us better error messages if the client is unable to connect to server for some reason.

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #430 from anmolnar/ZOOKEEPER-2893 and squashes the following commits:

aa735540 [Andor Molnar] ZOOKEEPER-2893. Use log4j message templates
47a8cf4c [Andor Molnar] ZOOKEEPER-2893. Make serverAddress local variable of run(). Separate SocketExceptions from generic ex handler and log at info level.
6ea4cb21 [Andor Molnar] ZOOKEEPER-2893. Renamed addr to serverAddress, use serverAddress in log message, it's always populated with the correct remote endpoint
fbe4ccde [Andor Molnar] ZOOKEEPER-2893. Make 'addr' variable available for error handling code to give a chance to fallback if the socket hasn't been initialized yet

Change-Id: I22becf9c1f923a28c82f263b604239fde9bc0ce4

7 months agoZOOKEEPER-2953: Flaky Test: testNoLogBeforeLeaderEstablishment
Abraham Fine [Sat, 16 Dec 2017 00:25:30 +0000 (16:25 -0800)] 
ZOOKEEPER-2953: Flaky Test: testNoLogBeforeLeaderEstablishment

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #432 from afine/ZOOKEEPER-2953 and squashes the following commits:

dc21603d [Abraham Fine] improve assertion logging
b5bc449c [Abraham Fine] use shared Servers and numServers
9fac0e20 [Abraham Fine] fix whitespace
68a03820 [Abraham Fine] ZOOKEEPER-2953: Flaky Test: testNoLogBeforeLeaderEstablishment

Change-Id: I8ee7b2d46783baa1e4cf2876cced740f4cd25ec2

7 months agoZOOKEEPER-2952: Upgrade third party libs: netty, slf4j
Andor Molnar [Wed, 13 Dec 2017 22:00:08 +0000 (14:00 -0800)] 
ZOOKEEPER-2952: Upgrade third party libs: netty, slf4j

https://issues.apache.org/jira/browse/ZOOKEEPER-2952

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #429 from anmolnar/ZOOKEEPER-2952 and squashes the following commits:

b488bf4b [Andor Molnar] ZOOKEEPER-2952. Updated slf4j license
949ada4a [Andor Molnar] ZOOKEEPER-2952. Upgrade third party libs: netty, slf4j

Change-Id: I7cce9cdbefab6c560a582c4dae7b4e00c73b1529

7 months agoZOOKEEPER-2950: Add keys for the Zxid from the stat command to check_zookeeper.py
Alex Bame [Tue, 12 Dec 2017 18:44:32 +0000 (10:44 -0800)] 
ZOOKEEPER-2950: Add keys for the Zxid from the stat command to check_zookeeper.py

Add keys for the zxid and its component pieces: epoch and transaction counter. These are not reported by the 'mntr' command so they must be obtained from 'stat'. The counter is useful for tracking transaction rates, and epoch is useful for tracking leader churn.

 zk_zxid         - the 64bit zxid from ZK
 zk_zxid_counter - the lower 32 bits, AKA the counter
 zk_zxid_epoch   - the upper 32 bits, AKA the epoch

Author: Alex Bame <alex@724support.com>

Reviewers: phunt@apache.org

Closes #425 from alexbb/add_zxid_stats and squashes the following commits:

8dfdc9fc [Alex Bame] leave zk_zxid in hex
874628db [Alex Bame] add keys for the Zxid from the stat command:  zk_zxid         - the 64bit zxid from ZK  zk_zxid_counter - the lower 32 bits, AKA the counter  zk_zxid_epoch   - the upper 32 bits, AKA the epoch

Change-Id: I5b43b8e27deab23669c7b44f777ddb02f50db4aa

7 months agoZOOKEEPER-2951: remove trailing \ from JAVA_HOME
Jorg Heymans [Tue, 12 Dec 2017 18:39:45 +0000 (10:39 -0800)] 
ZOOKEEPER-2951: remove trailing \ from JAVA_HOME

 fixes ZOOKEEPER-2951

Author: Jorg Heymans <jorg.heymans@gmail.com>

Reviewers: phunt@apache.org

Closes #431 from jorgheymans/master

Change-Id: Id47651ebdbb29574b47863640e928b6a99af6920

7 months agoZOOKEEPER-2338: set SOCK_CLOEXEC on socket if defined
Radu Brumariu [Wed, 6 Dec 2017 22:54:24 +0000 (14:54 -0800)] 
ZOOKEEPER-2338: set SOCK_CLOEXEC on socket if defined

Author: Radu Brumariu <radu@groupon.com>

Reviewers: phunt@apache.org

Closes #410 from fr0stbyte/ZOOKEEPER-2338

Change-Id: I7b060a2dc473b34aecc0cfac71d39720369bd635
(cherry picked from commit 508293d6a139eeb59d9dde5b4f72b7043f85350a)
Signed-off-by: Patrick Hunt <phunt@apache.org>
7 months agoZOOKEEPER-2924: Refactor tests of LoadFromLogTest.java
Andor Molnár [Wed, 6 Dec 2017 22:14:10 +0000 (14:14 -0800)] 
ZOOKEEPER-2924: Refactor tests of LoadFromLogTest.java

This patch is the trunk version of https://github.com/apache/zookeeper/pull/409

Author: Andor Molnár <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #420 from anmolnar/ZOOKEEPER-2924-trunk

Change-Id: I2260247e271eb86a6d37fe297e9fb5a7be3babc0

7 months agoZOOKEEPER-2935: [QP MutualAuth]: Port ZOOKEEPER-1045 implementation from branch-3...
Abraham Fine [Mon, 27 Nov 2017 23:10:13 +0000 (15:10 -0800)] 
ZOOKEEPER-2935: [QP MutualAuth]: Port ZOOKEEPER-1045 implementation from branch-3.5 to trunk

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #417 from afine/ZOOKEEPER-2935 and squashes the following commits:

99bfbe94 [Abraham Fine] Add debugging line and improve ivy.xml by removing unnecessary excludes
1d6c7de5 [Abraham Fine] Fix missing test.data.kerberos.dir in build.xml
06d0b6fa [Abraham Fine] ZOOKEEPER-2935: [QP MutualAuth]: Port ZOOKEEPER-1045 implementation from  branch-3.5 to trunk.

Change-Id: I2b88221c0006e4336a39f74fd5a87d1aded68c90

7 months agoZOOKEEPER-2948: Failing c unit tests on apache jenkins - fix the config.
Patrick Hunt [Wed, 22 Nov 2017 20:13:55 +0000 (12:13 -0800)] 
ZOOKEEPER-2948: Failing c unit tests on apache jenkins - fix the config.

Missed the quorum.cfg file which also needs to be updated for the new location.

Change-Id: Ifa65812c2b9c3e11b3f2137aa4d471964c5b4ad6

7 months agoZOOKEEPER-2948: Failing c unit tests on apache jenkins
Abraham Fine [Wed, 22 Nov 2017 17:41:34 +0000 (09:41 -0800)] 
ZOOKEEPER-2948: Failing c unit tests on apache jenkins

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #422 from afine/ZOOKEEPER-2948

Change-Id: I8d5d88d6b5863346dbb963f979ad9e6c9b33025c

7 months agoZOOKEEPER-2806: Flaky test: org.apache.zookeeper.server.quorum.FLEBackwardElectionRou...
Abraham Fine [Tue, 13 Jun 2017 18:57:22 +0000 (11:57 -0700)] 
ZOOKEEPER-2806: Flaky test: org.apache.zookeeper.server.quorum.FLEBackwardElectionRoundTest.testBackwardElectionRound

Change-Id: Ia92eb2b2dfbd9b7c464abdc7ae018f426e591538

7 months agoZOOKEEPER-2944: Specify correct overflow value
Chris Donati [Tue, 21 Nov 2017 17:44:24 +0000 (09:44 -0800)] 
ZOOKEEPER-2944: Specify correct overflow value

When a sequence counter exceeds 2147483647, the next value is -2147483648.

Author: Chris Donati <chris.donati@appscale.com>

Reviewers: phunt@apache.org

Closes #421 from cdonati/ZOOKEEPER-2944

Change-Id: If9ef0705e533ea9bc91d6da0af1984d5c93d89e5

7 months agoZOOKEEPER-2795: Change log level for "ZKShutdownHandler is not registered" error...
Abraham Fine [Tue, 21 Nov 2017 17:37:24 +0000 (09:37 -0800)] 
ZOOKEEPER-2795: Change log level for "ZKShutdownHandler is not registered" error message

`ZKShutdownHandler` may not be registered if the user creates a `ZooKeeperServer` object outside of `ZooKeeperServerMain.runFromConfig`.

We should change the log level of the message that is printed on state changes of `ZooKeeperServer` when `ZKShutdownHandler` is missing to something lower.

Author: Abraham Fine <afine@apache.org>

Reviewers: phunt@apache.org

Closes #273 from afine/ZOOKEEPER-2795

Change-Id: I8b048f6b79870064325925a6ff52f7d1f3e5e1da
(cherry picked from commit 86e35b5ae4e846e84dcc3ec08f724af7aa4ec2aa)
Signed-off-by: Patrick Hunt <phunt@apache.org>
8 months agoZOOKEEPER-2931: WriteLock recipe: Fix bug in znode ordering when the sessionId is...
Javier Cacheiro [Wed, 15 Nov 2017 23:21:33 +0000 (15:21 -0800)] 
ZOOKEEPER-2931: WriteLock recipe: Fix bug in znode ordering when the sessionId is part of the znode name

This PR solves a bug in the WriteLock recipe related to the ordering the znodes.

In the original version when the nodes are sorted in WriteLock.java using a TreeSet the whole znode path is taken into account and not just the sequence number.

This causes an issue when the sessionId is included in the znode path because a znode with a lower sessionId will appear as lower than other znode with a higher sessionId even if its sequence number is bigger. In specific situations this ended with two clients holding the lock at the same time.

Author: Javier Cacheiro <alcachi@gmail.com>

Reviewers: phunt@apache.org

Closes #413 from javicacheiro/fix_writelock and squashes the following commits:

ed0e6648 [Javier Cacheiro] Secondary sort znodes with the same sequence number by prefix
b382648d [Javier Cacheiro] Add unit tests
f9c581d4 [Javier Cacheiro] Fix bug in znode ordering when the sessionId is part of the znode name

Change-Id: Id384ce6079bbdfef10054fabb8bb93378d86024f

8 months agoZOOKEEPER-2934: Updated usage of LOG_DEBUG in recipes to follow changes in ZK client
Andor Molnár [Wed, 15 Nov 2017 22:08:13 +0000 (14:08 -0800)] 
ZOOKEEPER-2934: Updated usage of LOG_DEBUG in recipes to follow changes in ZK client

LOG_* macros have been changed recently in https://issues.apache.org/jira/browse/ZOOKEEPER-1400

This patch updates 'lock' and 'queue' recipes to follow the changes.

Author: Andor Molnár <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #416 from anmolnar/ZOOKEEPER-2934

Change-Id: I71fe3dcff3217c07c7c704723b7c3c1901975c6d

8 months agoZOOKEEPER-2923: The comment of the variable matchSyncs in class CommitProcessor has...
Jiafu Jiang [Wed, 15 Nov 2017 21:33:52 +0000 (13:33 -0800)] 
ZOOKEEPER-2923: The comment of the variable matchSyncs in class CommitProcessor has a mistake.

Author: Jiafu Jiang <jiangjiafu1989@gmail.com>

Reviewers: phunt@apache.org

Closes #408 from JiangJiafu/ZOOKEEPER-2923

Change-Id: Iaf5076b38aa950467b2664684ba98743ab131a5b
(cherry picked from commit 8be14c4d4743bebfd3e2b1759503d640480d4218)
Signed-off-by: Patrick Hunt <phunt@apache.org>
(cherry picked from commit c98d44a7af8e7883eb7e37d165b3ececb1f5cd7d)

8 months agoZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574
Mark Fenes [Wed, 15 Nov 2017 20:26:16 +0000 (12:26 -0800)] 
ZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574

ZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574

Added the documentation changes from PR #111 to the source (zookeeperAdmin.xml) and generated the new version of the html and pdf documents.

Note: I have not updated the 2nd paragraph from ZOOKEEPER-2574 as change "ZOOKEEPER-2349: Update documentation for snapCount" has a more recent version of that part of the text. (ZOOKEEPER-2349 was committed on Sep 11, 2017 while ZOOKEEPER-2574 was committed on Jan 23, 2017.)

Author: Mark Fenes <mfenes@cloudera.com>

Reviewers: phunt@apache.org

Closes #406 from mfenes/ZOOKEEPER-2690_master and squashes the following commits:

ba933658 [Mark Fenes] ZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574
b152f04d [Mark Fenes] ZOOKEEPER-2690: Update documentation source for ZOOKEEPER-2574

Change-Id: Ibfda946d7b25230308086969011fb2ecb10128d2

8 months agoy
Kfir Lev-Ari [Fri, 3 Nov 2017 04:04:56 +0000 (21:04 -0700)] 
y

We wish to fix this long-standing issue in the code.
Note that the previous commit processor algorithm had the same approach (as the one suggested in this fix) when dealing with a request that has a different cxid than session's expected one (see [here](https://github.com/apache/zookeeper/commit/9fc632c4f0a340b0a00ec6dff39c7b454c802822#diff-5cc688a027068714af01b0ad4d292fe5L238)).

This fix is based on the code from https://github.com/apache/zookeeper/pull/167, following the discussion in https://issues.apache.org/jira/browse/ZOOKEEPER-2684 .

Author: Kfir Lev-Ari <klevari@apple.com>

Reviewers: Alexander Shraer <shralex@apache.org>, Abraham Fine <afine@apache.org>

Closes #411 from kfirlevari/ZOOKEEPER-2684

8 months agoZOOKEEPER-2880 Rename README.txt to README.md - fixed packaging
Patrick Hunt [Wed, 1 Nov 2017 16:47:07 +0000 (09:47 -0700)] 
ZOOKEEPER-2880 Rename README.txt to README.md - fixed packaging

(cherry picked from commit 37e277162d567b55a07d1755f0b31c32e93c01a0)

9 months agoZOOKEEPER-1703: Added instructions on how to run the example code
Andor Molnar [Fri, 13 Oct 2017 23:12:59 +0000 (16:12 -0700)] 
ZOOKEEPER-1703: Added instructions on how to run the example code

Author: Andor Molnar <andor@cloudera.com>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #400 from dolphy17/master and squashes the following commits:

16e26f92 [Andor Molnar] ZOOKEEPER-1703: Set classpath the way that the book suggests
45ff9816 [Andor Molnar] ZOOKEEPER-1703: Added instructions on how to run the example code

Change-Id: I08467785ae3cdd75fd38c1e6d82439e4718d83fe

9 months agoZOOKEEPER-2915: Use "strict" conflict management in ivy
Abraham Fine [Wed, 11 Oct 2017 14:28:25 +0000 (07:28 -0700)] 
ZOOKEEPER-2915: Use "strict" conflict management in ivy

Author: Abraham Fine <afine@apache.org>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #397 from afine/ZOOKEEPER-2915

Change-Id: I7cb46c8aca7d30ba778d3d320a7d91da4bb2dc54

9 months agoZOOKEEPER-2890: Local automatic variable is left uninitialized and then freed.
xoiss [Tue, 10 Oct 2017 18:41:21 +0000 (11:41 -0700)] 
ZOOKEEPER-2890: Local automatic variable is left uninitialized and then freed.

Fixes https://issues.apache.org/jira/browse/ZOOKEEPER-2890

Author: xoiss <xoiss@ubuntu>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #359 from xoiss/branch-3.4-bugfix-zookeeper-2890

Change-Id: Ib3d132fc812a62b720967a888aff28edb7f3bdf1
(cherry picked from commit 948766de31eb955d62e661b7c55b74aa64434c77)
Signed-off-by: Patrick Hunt <phunt@apache.org>
9 months agoZOOKEEPER-2909: Create ant task to generate ivy dependency reports
Abraham Fine [Mon, 9 Oct 2017 17:57:13 +0000 (10:57 -0700)] 
ZOOKEEPER-2909: Create ant task to generate ivy dependency reports

Author: Abraham Fine <afine@apache.org>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #394 from afine/ZOOKEEPER-2909

Change-Id: Iecb30c5c0d722d81f7b10eaaa786bf80e4c37fe6

9 months agoZOOKEEPER-2914: compiler warning using java 9
Andor Molnar [Sat, 7 Oct 2017 19:22:42 +0000 (12:22 -0700)] 
ZOOKEEPER-2914: compiler warning using java 9

Replaced deprecated methods (Class.newInstance()) and removed redundant cast

Author: Andor Molnar <andor@cloudera.com>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #392 from dolphy17/ZOOKEEPER-2914 and squashes the following commits:

ab7bd5e2 [Andor Molnar] Reverted explicit case, because java 1.7 requires it to compile
b2d400bc [Andor Molnar] Replaced Class.newInstance() methods (deprecated in Java 9) with Constructor.newInstance()

Change-Id: Iae00f0874b69d425f35b96775d6ac9634b3ade73

9 months agoZOOKEEPER-2904: Remove unused imports from org.apache.zookeeper.server.quorum.WatchLe...
Nikhil Bhide [Wed, 4 Oct 2017 17:26:21 +0000 (10:26 -0700)] 
ZOOKEEPER-2904: Remove unused imports from org.apache.zookeeper.server.quorum.WatchLeakTest

Following imports are never used and unused imports do not adhere to code standard :

import org.apache.zookeeper.server.ZooTrace;
import java.util.ArrayList;
import java.util.List;

Author: Nikhil Bhide <nikhilbhide15@gmail.com>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #387 from nikhilbhide/ZOOKEEPER-2904

Change-Id: Ib1140021d3432c9e7981dfdf48e75ea1a2de325f

9 months agoZOOKEEPER-2906: The OWASP dependency check jar should not be included in the default...
Abraham Fine [Fri, 29 Sep 2017 22:53:40 +0000 (15:53 -0700)] 
ZOOKEEPER-2906: The OWASP dependency check jar should not be included in the default classpath in the default classpath

Author: Abraham Fine <afine@apache.org>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #384 from afine/ZOOKEEPER-2906 and squashes the following commits:

45e8eb98 [Abraham Fine] Move releaseaudit jars to their own directory
241fbeb2 [Abraham Fine] ZOOKEEPER-2906: The OWASP dependency check jar should not be included in the default classpath

Change-Id: I6267ddbe15fd932f20ab7fbaa4909e03960c5497

9 months agoZOOKEEPER-2887: define dependency versions in build.xml to be easily overridden in...
Tamas Penzes [Thu, 28 Sep 2017 03:09:44 +0000 (20:09 -0700)] 
ZOOKEEPER-2887: define dependency versions in build.xml to be easily overridden in build.properties

If the dependency versions are defined in build.xml they can be easily
overridden by re-defining them in build.properties
This process can be useful to avoid classpath clashes among different
Hadoop components

Author: Tamas Penzes <tamaas@cloudera.com>
Author: Tamás Pénzes <penzes.tamas@gmail.com>

Reviewers: Patrick Hunt <phunt@apache.org>, Michael Han <hanm@apache.org>

Closes #357 from tamaashu/ZOOKEEPER-2887 and squashes the following commits:

5e2f43fb [Tamás Pénzes] Merge branch 'master' into ZOOKEEPER-2887
65558c09 [Tamas Penzes] ZOOKEEPER-2887: define dependency versions in build.xml to be easily overridden in build.properties
6cf315fd [Tamas Penzes] ZOOKEEPER-2887: define dependency versions in build.xml to be easily overridden in build.properties

Change-Id: Ic19d67cc68ed4595ae0636d8200287400e98e2a1

9 months agoZOOKEEPER-2905: Don't include `config.h` in `zookeeper.h`
Andrew Schwartzmeyer [Wed, 27 Sep 2017 22:18:15 +0000 (15:18 -0700)] 
ZOOKEEPER-2905: Don't include `config.h` in `zookeeper.h`

In ZOOKEEPER-2841 I fixed the inclusion of project-specific porting code
that were included in the public headers, which then broke upstream
projects (in my case, Mesos).

Unfortunately, I inadvertently created a very similar problem, and it
wasn't evident until the build was coupled with another project with the
same bug. More specifically, when including ZooKeeper in Mesos, and
including Google's Glog in Mesos, both projects define the macros
`VERSION`, `PACKAGE_VERSION`, and `PACKAGE_TARNAME`, and do so publicly.
This is commonly defined in `config.h` by Autotools (and by CMake for
ZooKeeper for compatibility), and is not a problem unless included
publicly, such as in `zookeeper.h`, and by more than one project.

When refactoring, I saw two includes in `zookeeper.h` that instead of
being guarded by e.g. `#ifdef HAVE_SYS_SOCKET_H` were guarded by
`#ifndef WIN32`. I erroneously added `#include "config.h"` and guarded
the includes "properly" with a feature guard. However, configuration
files such as `config.h` and `winconfig.h` etc. must never be included
in publicly in `zookeeper.h`, for the reasons given above.

This patch reverts the bug, and instead includes `config.h` in
`zookeeper.c`, where it is not exposed to other projects.

Author: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #383 from andschwa/ZOOKEEPER-2905-master

Change-Id: I7c94bbf9dd8a4bd0548fbf07cbb3ff1f0c3bbe06

9 months agoZOOKEEPER-2896: Remove unused imports from org.apache.zookeeper.test.CreateTest.java
Nikhil Bhide [Wed, 27 Sep 2017 21:27:51 +0000 (14:27 -0700)] 
ZOOKEEPER-2896: Remove unused imports from org.apache.zookeeper.test.CreateTest.java

Following imports are not used in the code and do not adhere to code convention and style.

    import java.util.ArrayList;
    import java.util.Collections;
    import java.util.List;
    import org.apache.zookeeper.AsyncCallback.Create2Callback;

Author: Nikhil Bhide <nikhilbhide15@gmail.com>

Reviewers: Patrick Hunt <phunt@apache.org>

Closes #379 from nikhilbhide/ZOOKEEPER-2896

Change-Id: I49aea37755d8f8f7e4a70e4ecf33e8b069239ee9

10 months agoZOOKEEPER-2777: There is a typo in zk.py which prevents from using/compiling it
Nikhil Bhide [Mon, 11 Sep 2017 21:29:22 +0000 (14:29 -0700)] 
ZOOKEEPER-2777: There is a typo in zk.py which prevents from using/compiling it

Author: Nikhil Bhide <nikhilbhide15@gmail.com>

Reviewers: Frederic Leger, Michael Han <hanm@apache.org>

Closes #367 from nikhilbhide/ZOOKEEPER-2777

10 months agoZOOKEEPER-2581: Not handled NullPointerException while creating key manager and trust...
maoling [Mon, 11 Sep 2017 20:56:46 +0000 (13:56 -0700)] 
ZOOKEEPER-2581: Not handled NullPointerException while creating key manager and trustManager

- when **zookeeper.ssl.keyStore.password**    is null,**keyStorePasswordProp** is **null** ,then **keyStorePassword.toCharArray()** is called,it will throw a NPE. we should handle this NPE gracefully
- I just follow an example from [createSSLContext](https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/common/X509Util.java#L87)

Author: maoling <maoling199210191@sina.com>

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Michael Han <hanm@apache.org>

Closes #339 from maoling/ZOOKEEPER-2581

10 months agoZOOKEEPER-2349: Update documentation for snapCount
maoling [Mon, 11 Sep 2017 20:43:11 +0000 (13:43 -0700)] 
ZOOKEEPER-2349: Update documentation for snapCount

Origin patch can not apply now and it has something inaccurate.so I give it a new PR.
more details in [JIRA](https://issues.apache.org/jira/browse/ZOOKEEPER-2349)

Author: maoling <maoling199210191@sina.com>

Reviewers: Abe Fine <afine@apache.org>, Michael Han <hanm@apache.org>

Closes #349 from maoling/ZOOKEEPER-2349

10 months agoZOOKEEPER-2630: Use interface type instead of implementation type whe…
Tamas Penzes [Mon, 11 Sep 2017 16:55:05 +0000 (09:55 -0700)] 
ZOOKEEPER-2630: Use interface type instead of implementation type whe…

Use interface type instead of implementation type when appropriate.

There are a couple of places in code base where we declare a field / variable as implementation type (i.e. HashMap, HashSet) instead of interface type (i.e. Map, Set), while in other places we do the opposite by declaring as interface type. A quick check indicates that most if not all of these places could be updated so we have a consistent style over the code base (prefer using interface type), which is also a good coding style to stick per best practice.

Checked and fixed Set, Map and List interface usages.

Author: Tamas Penzes <tamaas@cloudera.com>

Reviewers: Abe Fine <afine@apache.org>, Michael Han <hanm@apache.org>

Closes #354 from tamaashu/ZOOKEEPER-2630

10 months agoZOOKEEPER-2875: Add ant task for running OWASP dependency report
Abraham Fine [Mon, 11 Sep 2017 04:33:29 +0000 (21:33 -0700)] 
ZOOKEEPER-2875: Add ant task for running OWASP dependency report

Simply run `ant owasp` and a report will be placed in `build/test/owasp`. The task will return a nonzero status code if there are any vulnerabilities in any of the dependencies.

Author: Abraham Fine <afine@apache.org>

Reviewers: Michael Han <hanm@apache.org>

Closes #340 from afine/ZOOKEEPER-2875