Skip to content
This repository was archived by the owner on May 21, 2020. It is now read-only.

Commit 4267c60

Browse files
aserralleriosaserrallerios
committed
Import source from alpakka
1 parent 65f5050 commit 4267c60

24 files changed

+1278
-1
lines changed

‎.gitignore

+19-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,20 @@
1-
*.class
1+
.idea*
2+
*.env
23
*.log
4+
*.iml
5+
target/
6+
.target/
7+
.DS_Store
8+
.cache*
9+
.classpath
10+
.project
11+
.settings
12+
.tmpBin/
13+
*.sublime-project
14+
/bin/
15+
ext-lib-src/
16+
.ensime
17+
.ensime_cache/
18+
moquette_store.mapdb
19+
moquette_store.mapdb.p
20+
moquette_store.mapdb.t

‎.travis.yml

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
language: scala
2+
3+
sudo: false
4+
5+
scala:
6+
- "2.11.11"
7+
- "2.12.2"
8+
9+
jdk:
10+
- oraclejdk8
11+
12+
script:
13+
- sbt -J-XX:ReservedCodeCacheSize=128m ++$TRAVIS_SCALA_VERSION ";test:compile"
14+
# make 'git branch' work again
15+
- git branch -f "$TRAVIS_BRANCH" && git checkout "$TRAVIS_BRANCH"
16+
# make comparing to origin/master work
17+
- git remote set-branches --add origin master
18+
- git fetch
19+
# check unformatted code
20+
- git diff --exit-code --color || { echo "[error] Unformatted code found. Please run 'test:compile' and commit the reformatted code."; false; }
21+
22+
before_cache:
23+
- find $HOME/.ivy2 -name "ivydata-*.properties" -print -delete
24+
- find $HOME/.sbt -name "*.lock" -print -delete
25+
26+
cache:
27+
directories:
28+
- $HOME/.ivy2/cache
29+
- $HOME/.sbt/boot
30+
31+
deploy:
32+
provider: script
33+
skip_cleanup: true
34+
script:
35+
- sbt -J-XX:ReservedCodeCacheSize=128m ++$TRAVIS_SCALA_VERSION publish
36+
on:
37+
tags: true
38+
repo: akka/alpakka
39+
condition: $AKKA_SERIES = 2.4
40+
41+
env:
42+
matrix:
43+
- AKKA_SERIES=2.4
44+
- AKKA_SERIES=2.5
45+
global:
46+
# encrypt with: travis encrypt BINTRAY_USER=...
47+
- secure: "foo"
48+
# encrypt with: travis encrypt BINTRAY_PASS=...
49+
- secure: "foo"

‎LICENSE

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
This software is licensed under the Apache 2 license, quoted below.
2+
3+
Copyright 2018 Albert Serrallé
4+
5+
Licensed under the Apache License, Version 2.0 (the "License"); you may not
6+
use this file except in compliance with the License. You may obtain a copy of
7+
the License at
8+
9+
[http://www.apache.org/licenses/LICENSE-2.0]
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
13+
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
14+
License for the specific language governing permissions and limitations under
15+
the License.

‎README.md

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Akka Stream Source for Kinesis Client Library [![travis-badge][]][travis]
2+
3+
[travis]: https://travis-ci.org/aserrallerios/kcl-akka-stream
4+
[travis-badge]: https://travis-ci.org/aserrallerios/kcl-akka-stream.svg?branch=master
5+
6+
For more information about Kinesis please visit the [official documentation](https://aws.amazon.com/documentation/kinesis/).
7+
8+
The KCL Source can read from several shards and rebalance automatically when other Workers are started or stopped. It also handles record sequence checkpoints.
9+
10+
For more information about KCL please visit the [official documentation](http://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html).
11+
12+
## Installation
13+
14+
TODO bintray publish
15+
16+
## Usage
17+
18+
### AWS KCL Worker Source & checkpointer
19+
20+
The KCL Worker Source needs to create and manage Worker instances in order to consume records from Kinesis Streams.
21+
22+
In order to use it, you need to provide a Worker builder and the Source settings:
23+
24+
```scala
25+
val workerSourceSettings = KinesisWorkerSourceSettings(
26+
bufferSize = 1000,
27+
checkWorkerPeriodicity = 1 minute)
28+
val builder: IRecordProcessorFactory => Worker = { recordProcessorFactory =>
29+
new Worker.Builder()
30+
.recordProcessorFactory(recordProcessorFactory)
31+
.config(
32+
new KinesisClientLibConfiguration(
33+
"myApp",
34+
"myStreamName",
35+
DefaultAWSCredentialsProviderChain.getInstance(),
36+
s"${
37+
import scala.sys.process._
38+
"hostname".!!.trim()
39+
}:${java.util.UUID.randomUUID()}"
40+
)
41+
)
42+
.build()
43+
}
44+
```
45+
46+
The Source also needs an `ExecutionContext` to run the Worker's thread and to execute record checkpoints. Then the Source can be created as usual:
47+
48+
```scala
49+
implicit val _ =
50+
ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(1000))
51+
KinesisWorker(builder, workerSourceSettings).to(Sink.ignore)
52+
```
53+
54+
### Committing records
55+
56+
The KCL Worker Source publishes messages downstream that can be committed in order to mark progression of consumers by shard. This process can be done manually or using the provided checkpointer Flow.
57+
58+
In order to use the Flow you can provide additional settings:
59+
60+
```scala
61+
val checkpointSettings = KinesisWorkerCheckpointSettings(100, 30 seconds)
62+
KinesisWorker(builder, workerSourceSettings)
63+
.via(KinesisWorker.checkpointRecordsFlow(checkpointSettings))
64+
.to(Sink.ignore)
65+
KinesisWorker(builder, workerSourceSettings).to(
66+
KinesisWorker.checkpointRecordsSink(checkpointSettings))
67+
```
68+
69+
## License
70+
71+
Copyright (c) 2018 Albert Serrallé
72+
73+
This version of *kcl-akka-stream* is released under the Apache License, Version 2.0 (see LICENSE.txt).
74+
By downloading and using this software you agree to the
75+
[End-User License Agreement (EULA)](LICENSE).
76+
77+
We build on a number of third-party software tools, with the following licenses:
78+
79+
#### Java Libraries
80+
81+
Third-Party software | License
82+
----------------------------|-----------------------
83+
amazon-kinesis-client | Amazon Software License

‎build.sbt

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
lazy val `kcl-akka-stream` =
2+
Project(id = "kcl-akka-stream", base = file("."))
3+
.enablePlugins(AutomateHeaderPlugin)
4+
.settings(
5+
name := "kcl-akka-stream",
6+
// By default scalatest futures time out in 150 ms, dilate that to 600ms.
7+
// This should not impact the total test time as we don't expect to hit this
8+
// timeout.
9+
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest,
10+
"-F",
11+
"4")
12+
)
13+
.settings(
14+
Dependencies.Kinesis,
15+
// For mockito
16+
parallelExecution in Test := false,
17+
onLoadMessage :=
18+
"""
19+
|** Welcome to the sbt build definition for kcl-akka-stream! **
20+
""".stripMargin
21+
)

‎project/Common.scala

+52
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import sbt._
2+
import sbt.Keys._
3+
import sbt.plugins.JvmPlugin
4+
import com.lucidchart.sbt.scalafmt.ScalafmtCorePlugin.autoImport._
5+
import de.heikoseeberger.sbtheader._
6+
import de.heikoseeberger.sbtheader.HeaderPlugin.autoImport._
7+
import sbtunidoc.BaseUnidocPlugin.autoImport._
8+
9+
object Common extends AutoPlugin {
10+
11+
override def trigger = allRequirements
12+
13+
override def requires = JvmPlugin && HeaderPlugin
14+
15+
override lazy val projectSettings =
16+
Dependencies.Common ++ Seq(
17+
// organization := "com.lightbend.akka",
18+
// organizationName := "Lightbend Inc.",
19+
homepage := Some(url("https://github.com/aserrallerios/kcl-akka-stream")),
20+
scmInfo := Some(ScmInfo(url("https://github.com/aserrallerios/kcl-akka-stream"), "[email protected]:aserrallerios/kcl-akka-stream.git")),
21+
developers += Developer("aserralle",
22+
"Albert Serrallé",
23+
"[email protected]",
24+
url("https://github.com/aserrallerios")),
25+
licenses := Seq(("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0"))),
26+
crossVersion := CrossVersion.binary,
27+
scalacOptions ++= Seq(
28+
"-encoding",
29+
"UTF-8",
30+
"-feature",
31+
"-unchecked",
32+
"-deprecation",
33+
//"-Xfatal-warnings",
34+
"-Xlint",
35+
"-Yno-adapted-args",
36+
"-Ywarn-dead-code",
37+
"-Xfuture"
38+
),
39+
javacOptions in compile ++= Seq(
40+
"-Xlint:unchecked"
41+
),
42+
// autoAPIMappings := true,
43+
// apiURL := Some(url()),
44+
// show full stack traces and test case durations
45+
testOptions in Test += Tests.Argument("-oDF"),
46+
// -v Log "test run started" / "test started" / "test run finished" events on log level "info" instead of "debug".
47+
// -a Show stack traces and exception class name for AssertionErrors.
48+
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
49+
scalafmtOnCompile := true,
50+
headerLicense := Some(HeaderLicense.Custom("Copyright (C) 2018 Albert Serrallé"))
51+
)
52+
}

‎project/Dependencies.scala

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import sbt._, Keys._
2+
3+
object Dependencies {
4+
5+
val AkkaVersion = sys.env.get("AKKA_SERIES") match {
6+
case Some("2.5") => "2.5.6"
7+
case _ => "2.4.19"
8+
}
9+
10+
val AwsSdkVersion = "1.11.226"
11+
12+
val Common = Seq(
13+
libraryDependencies ++= Seq(
14+
"com.typesafe.akka" %% "akka-stream" % AkkaVersion,
15+
"com.typesafe.akka" %% "akka-stream-testkit" % AkkaVersion % Test,
16+
"org.scalatest" %% "scalatest" % "3.0.1" % Test, // ApacheV2
17+
"com.novocode" % "junit-interface" % "0.11" % Test, // BSD-style
18+
"junit" % "junit" % "4.12" % Test // Eclipse Public License 1.0
19+
)
20+
)
21+
22+
val Kinesis = Seq(
23+
libraryDependencies ++= Seq(
24+
"com.amazonaws" % "aws-java-sdk-kinesis" % AwsSdkVersion, // ApacheV2
25+
"com.amazonaws" % "amazon-kinesis-client" % "1.8.8", // Amazon Software License
26+
"org.mockito" % "mockito-core" % "2.7.11" % Test // MIT
27+
)
28+
)
29+
}

‎project/Publish.scala

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
import sbt._, Keys._
2+
3+
/**
4+
* For projects that are not to be published.
5+
*/
6+
object NoPublish extends AutoPlugin {
7+
override def requires = plugins.JvmPlugin
8+
9+
override def projectSettings = Seq(
10+
publishArtifact := false,
11+
publish := {},
12+
publishLocal := {}
13+
)
14+
}
15+
16+
object Publish extends AutoPlugin {
17+
import bintray.BintrayPlugin
18+
import bintray.BintrayPlugin.autoImport._
19+
20+
override def trigger = allRequirements
21+
override def requires = BintrayPlugin
22+
23+
override def projectSettings = Seq(
24+
bintrayOrganization := Some("aserralle"),
25+
bintrayPackage := "kcl-akka-stream"
26+
)
27+
}
28+
29+
object PublishUnidoc extends AutoPlugin {
30+
import sbtunidoc.BaseUnidocPlugin._
31+
import sbtunidoc.BaseUnidocPlugin.autoImport._
32+
import sbtunidoc.ScalaUnidocPlugin.autoImport.ScalaUnidoc
33+
34+
override def requires = sbtunidoc.ScalaUnidocPlugin
35+
36+
def publishOnly(artifactType: String)(config: PublishConfiguration) = {
37+
val newArts = config.artifacts.filter(_._1.`type` == artifactType)
38+
config.withArtifacts(newArts)
39+
}
40+
41+
override def projectSettings = Seq(
42+
doc in Compile := (doc in ScalaUnidoc).value,
43+
target in unidoc in ScalaUnidoc := crossTarget.value / "api",
44+
publishConfiguration ~= publishOnly(Artifact.DocType),
45+
publishLocalConfiguration ~= publishOnly(Artifact.DocType)
46+
)
47+
}

‎project/TestChanged.scala

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
package akka.stream.alpakka
2+
3+
import scala.collection.immutable
4+
import scala.sys.process._
5+
6+
import sbt._
7+
import sbt.Keys._
8+
9+
object TestChanged extends AutoPlugin {
10+
override def trigger = allRequirements
11+
override def requires = plugins.JvmPlugin
12+
13+
val changedDirectories = taskKey[immutable.Set[String]]("List of touched modules in this PR branch")
14+
val testChanged = taskKey[Unit]("Test all subprojects with changes compared to master")
15+
16+
override lazy val buildSettings = Seq(
17+
changedDirectories := {
18+
val log = streams.value.log
19+
val target = "origin/master"
20+
21+
// TODO could use jgit
22+
val diffOutput = s"git diff $target --name-only".!!.split("\n")
23+
val changedDirectories =
24+
diffOutput
25+
.map(l => l.trim)
26+
.map(l ⇒ l.takeWhile(_ != '/'))
27+
.map(new File(_))
28+
.map(file => if (file.isDirectory) file.toString else "")
29+
.toSet
30+
31+
log.info("Detected changes in directories: " + changedDirectories.mkString("[", ", ", "]"))
32+
changedDirectories
33+
}
34+
)
35+
36+
override lazy val projectSettings = Seq(
37+
testChanged := Def.taskDyn {
38+
val skip = Def.setting { task(()) }
39+
if (shouldBuild(name.value, changedDirectories.value)) test in Test
40+
else skip
41+
}.value
42+
)
43+
44+
implicit class RegexHelper(val sc: StringContext) extends AnyVal {
45+
def re: scala.util.matching.Regex = sc.parts.mkString.r
46+
}
47+
48+
private def shouldBuild(projectName: String, changedDirectories: Set[String]) = projectName match {
49+
case "alpakka" => false
50+
case re"akka-stream-alpakka-(.+)$subproject" =>
51+
changedDirectories.contains(subproject) || changedDirectories.contains("") || changedDirectories.contains(
52+
"project"
53+
)
54+
}
55+
}

0 commit comments

Comments
 (0)