SAMZA-2757: Make Samza Compatible with Java 11 #1628

james-deee · 2022-08-18T21:13:08Z

Issues: The current Samza Project does not support running in a Java 11 runtime environment. It will continue to build this project with Java 8, however.

Changes:

Update the README to accurately reflect how this project now operates with Java 8 and Java 11.
Changes some required things to the run-class to get it to run in a Java 11 environment.
Adds a new samza-yarn3 and samza-shell-yarn3 project to support people wanting to run in Java 11 environments with the required Yarn version of 3.3.+ (which is when Yarn started supporting Java 11).
Upgrades Gradle to V6, everything seems to work fine with the upgrade
Various import changes moving to org.apache.commons.lang3

Tests:

Ran all tests in this project using both Java 11 and Java 8.
Locally published these changes and ran with the updated Hello Samza application with both Java 8 and Java 11 using Yarn 3.3.4 and the samza-yarn3 module.

API Changes: None

Upgrade Instructions: All current users of Samza should be able to continue using Samza as usual, there is nothing required for upgrading to this new version.

Usage Instructions: If you want to use Java 11, then you should make use of the samza-yarn3 and samza-shell-yarn3 modules. The following modules do NOT support Yarn 3:

samza-test
samza-hdfs
samza-yarn

samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala

samza-shell/src/main/bash/run-class.sh

james-deee · 2022-09-02T18:57:56Z

@xinyuiscool Sorry to tag you, but I see you were the last one to merge something to this project. Would you be able to take a look at this PR, and the corresponding hello samza one? apache/samza-hello-samza#87

…za-to-build-with-java11-real

…james-deee/samza into jtd/get-samza-to-build-with-java11-real

Make changes for some failing issues in tests with Java 11

…version

Issues: Upgrade dependencies for security fixes and enhancements for jetty and jackson In current dependencies there are security vulnerabilities Jackson: https://security.snyk.io/package/maven/com.fasterxml.jackson.core:jackson-databind/2.12.2 Jetty: https://security.snyk.io/package/maven/org.eclipse.jetty:jetty-server/9.4.38.v20210224 Upgrade Jackson to version: 2.13.3 Upgrade Jetty to 9.4.48.v20220622

…pache#1627) Issues: Currently, StreamAppender is not extensible in terms of setting systemProducer and exposing sendEventToSystemProducer to child classes. It also makes it hard for child classes out of package to unit test any logic related to systemProducer Changes: Make StreamAppender extensible 1. Change method sendEventToSystemProducer from private to protected 2. Change member variable systemProducer from private to protected

samza-yarn3/src/test/scala/org/apache/samza/job/yarn/TestSamzaYarnAppMasterLifecycle.scala

nickpan47

Mostly lgtm. Please address the few minor comments. Thanks a lot!

README.md

gradle.properties

samza-log4j2/src/main/java/org/apache/samza/logging/log4j2/StreamAppender.java

samza-shell/src/main/bash/run-class.sh

samza-yarn/src/test/java/org/apache/samza/job/yarn/TestYarnFaultDomainManager.java

samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaYarnAppMasterLifecycle.scala

…za-to-build-with-java11-real

…the -XX:+PrintGCDateStamps flag

…a/job/yarn/TestYarnFaultDomainManager.java

nickpan47

The corresponding samza-shell-yarn3 module does not yield a tgz under samza-shell-yarn3/build/distribution directory. Please see my review comments.

nickpan47 · 2022-10-06T04:59:41Z

build.gradle

+  task shellTarGz(type: Tar) {
+    compression = Compression.GZIP
+    classifier = 'dist'
+    from 'src/main/bash'


Are you sure that this works? I pulled down the PR and built locally. Only found that the corresponding samza-shell-yarn3 build does not yield any tgz file as output in build/distribution, since there is no separate samza-shell-yarn3/src in the source repo.

Made the following change here and the build command successfully generates the samza-shell-yarn3 .tgz file:

`

from 'src/main/bash'

from 'src/main/resources'

from 'src/main/visualizer'

from(project(":samza-shell").file("./src/main/bash"))

from(project(":samza-shell").file("./src/main/resources"))

from(project(":samza-shell").file("./src/main/visualizer"))
`

Oh jeez. what a dummy i am. Should probably make sure the actual distribution get's build 🤦 .

Thanks! I updated it.

nickpan47

LGTM. Thanks for your contribution!

Feature: Make Samza build artifacts runnable in JDK11 Changes: - introduced new build modules samza-yarn3 and samza-shell-yarn3 to build Samza w/ YARN 3.3.4, which is runtime compatible w/ JDK11  Tests: - local test and hello-samza succeeded with Java11

This reverts commit 0396739.

Feature: Make Samza build artifacts runnable in JDK11 Changes: - introduced new build modules samza-yarn3 and samza-shell-yarn3 to build Samza w/ YARN 3.3.4, which is runtime compatible w/ JDK11  Tests: - local test and hello-samza succeeded with Java11

This reverts commit 5580bcf. RB=3722433 G=stream-processing-reviewers A=xiliu,alazhang

james-deee and others added 4 commits August 18, 2022 11:11

Initial commit for getting Samza to work with Java 11

5d6e588

Upgrade Hadoop to 3.3.4

1dbd8a0

Fix test

e67cc59

Update gradle.properties

332ca88

james-deee commented Aug 18, 2022

View reviewed changes

samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala Show resolved Hide resolved

james-deee commented Aug 18, 2022

View reviewed changes

samza-shell/src/main/bash/run-class.sh Show resolved Hide resolved

james-deee changed the title ~~Get Samza to work with Java 11 (still works with Java 8)~~ SAMZA-2757 : Make Samza Compatible with Java 11 Aug 19, 2022

james-deee added 4 commits September 17, 2022 07:46

Merge branch 'master' of github.com:james-deee/samza into jtd/get-sam…

c38e2d9

…za-to-build-with-java11-real

Merge branch 'jtd/get-samza-to-build-with-java11-real' of github.com:…

f2971be

…james-deee/samza into jtd/get-samza-to-build-with-java11-real

Move yarn back to 2.10.1 by default.

2cd5b5a

Make changes for some failing issues in tests with Java 11

Upgrade gradle, remove yarn 3 change

380486d

james-deee closed this Sep 21, 2022

james-deee reopened this Sep 21, 2022

james-deee and others added 3 commits September 21, 2022 10:37

Make a samza-yarn3 project that can be included given the right yarn …

bb5080b

…version

james-deee force-pushed the jtd/get-samza-to-build-with-java11-real branch from 0d9bc04 to 8058a51 Compare September 21, 2022 15:38

james-deee commented Sep 21, 2022

View reviewed changes

samza-yarn3/src/test/scala/org/apache/samza/job/yarn/TestSamzaYarnAppMasterLifecycle.scala Show resolved Hide resolved

Apply patch from Yi

540a528

nickpan47 requested changes Sep 29, 2022

View reviewed changes

james-deee added 8 commits October 3, 2022 12:05

PR comments and revert of powermock changes

cad892e

Merge branch 'master' of github.com:james-deee/samza into jtd/get-sam…

6d24d92

…za-to-build-with-java11-real

Another powermock revert and adding in conditional to run script for …

5545325

…the -XX:+PrintGCDateStamps flag

A couple more reverts

5d9c790

Revert the yarn2 changes for samza-yarn/src/test/java/org/apache/samz…

bcddf68

…a/job/yarn/TestYarnFaultDomainManager.java

Remove commented out lines

a18e6d1

missed backtick

473d39b

Add a samza-shell-yarn3 module

d1760cf

nickpan47 self-requested a review October 5, 2022 18:38

nickpan47 requested changes Oct 6, 2022

View reviewed changes

Get the actual distribution to be built 🤦

50e1baf

james-deee changed the title ~~SAMZA-2757 : Make Samza Compatible with Java 11~~ SAMZA-2757: Make Samza Compatible with Java 11 Oct 7, 2022

nickpan47 self-requested a review October 8, 2022 04:58

nickpan47 approved these changes Oct 8, 2022

View reviewed changes

nickpan47 merged commit 5580bcf into apache:master Oct 10, 2022

ehoner pushed a commit to ehoner/samza that referenced this pull request Apr 11, 2023

Revert "SAMZA-2757: Make Samza Compatible with Java 11 (apache#1628)"

9b77be6

This reverts commit 0396739.

ehoner pushed a commit to ehoner/samza that referenced this pull request Apr 11, 2023

Revert "SAMZA-2757: Make Samza Compatible with Java 11 (apache#1628)"

411ecb0

This reverts commit 5580bcf. RB=3722433 G=stream-processing-reviewers A=xiliu,alazhang

SAMZA-2757: Make Samza Compatible with Java 11 #1628

SAMZA-2757: Make Samza Compatible with Java 11 #1628

Uh oh!

Conversation

james-deee commented Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-deee commented Sep 2, 2022

Uh oh!

Uh oh!

nickpan47 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nickpan47 left a comment

Choose a reason for hiding this comment

Uh oh!

nickpan47 Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

nickpan47 Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

james-deee Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

nickpan47 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

james-deee commented Aug 18, 2022 •

edited

Loading