Introduction
In March 2019 Shady and I visited Voxxed Days Romania in Bucharest. If you haven’t seen our post about that, check it out now! There were some really cool talks and so I decided to pick one and write about it.
At my previous employer, we switched from a monolithic service to a microservice architecture. After implementing about 20 different microservices in 2 years, the communication between them got more complex. In addition to that, all microservices where communicating synchronously! Did we build another monolith? I just recently read a blog post about that on another site: https://thenewstack.io/synchronous-rest-turns-microservices-back-monoliths/
So back to the topic… This is why I always was interested in asynchronous communication (streams, message bus, pubsub, whatever). I heard a lot from Uber using Google Clouds PubSub, how it’s highly scalable, asynchronous and most important: just cool to use! I was inspired by Mark Heckler’s talk “Drinking from the Stream: How to Use Messaging Platforms for Scalability&Performance” and tried it out myself. Of course, I’m sharing my experiences and example with you…
Technologies
Spring Cloud Stream
Spring Cloud Stream is a framework for building highly scalable event-driven microservices connected with shared messaging systems.
https://spring.io/projects/spring-cloud-stream#overview
Spring Cloud Stream supports a variety of binder implementations:
- RabbitMQ
- Apache Kafka
- Kafka Streams
- Amazon Kinesis
- Google PubSub (partner maintained)
- Solace PubSub+ (partner maintained)
- Azure Event Hubs (partner maintained)
We will use Spring Cloud Stream to create 3 different projects (microservices), with the Apache Kafka Binder using the Spring Initializr.
Documentation
Kafka
Apache Kafka is a distributed streaming platform. Communication between endpoints is driven by messaging-middleware parties like Apache Kafka or RabbitMQ.
Documentation
https://kafka.apache.org/documentation/
Let’s get started!
Prerequisites
So this is all you need to get yourself started:
- Maven 3.2+
- Java 7+ (Java 8 highly recommended!)
- Docker
The idea: Money money money 💰
Let’s build a money-printing machine 🤑! So the idea is…
- Producer
- Prints money (coins and notes) in different currencies, values and qualities.
- Processor
- Fetch money and polish coins/notes to”perfect” quality. This is quality assurance 😉.
- Consumer
- Fetch (spend) money and show type, currency, value and quality.
Bootstrap your application with Spring Initializr
Create a new project just with a few clicks 🖱
- Project: Maven Project
- Language: Java
- Spring Boot: 2.1.4
- Project Metadata
- Group: com.47northlabs
- Artefact: moneyprinter-producer
- Dependencies
- Web
- Cloud Stream
- Kafka
- Lombok
Implementation of the producer
Create or edit /src/main/resources/application.properties
server.port=0
spring.cloud.stream.bindings.output.destination=processor
spring.cloud.stream.bindings.output.group=processor
spring.cloud.stream.kafka.binder.auto-add-partitions=true
spring.cloud.stream.kafka.binder.min-partition-count=4
The destination defines to which pipeline (or topic) the message is published to.
Create or edit /src/main/java/com/northlabs/lab/moneyprinterproducer/MoneyprinterProducerApplication.java
package com.northlabs.lab.moneyprinterproducer;
import lombok.AllArgsConstructor;
import lombok.Data;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.stream.annotation.EnableBinding;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.messaging.support.MessageBuilder;
import java.util.Random;
import java.util.UUID;
@SpringBootApplication
public class MoneyprinterProducerApplication {
public static void main(String[] args) {
SpringApplication.run(MoneyprinterProducerApplication.class, args);
}
}
@EnableBinding(Source.class)
@EnableScheduling
@AllArgsConstructor
class Spammer {
private final Source source;
private final SubscriberGenerator generator;
@Scheduled(fixedRate = 1000)
private void spam() {
Money money = generator.printMoney();
System.out.println(money);
source.output().send(MessageBuilder.withPayload(money).build());
}
}
@Component
class SubscriberGenerator {
private final String[] type = "Coin, Note".split(", ");
private final String[] currency = "CHF, EUR, USD, JPY, GBP".split(", ");
private final String[] value = "1, 2, 5, 10, 20, 50, 100, 200, 500, 1000".split(", ");
private final String[] quality = "poor, fair, good, premium, flawless, perfect".split(", ");
private final Random rnd = new Random();
private int i = 0, j = 0, k=0, l=0;
Money printMoney() {
i = rnd.nextInt(2);
j = rnd.nextInt(5);
k = rnd.nextInt(10);
l = rnd.nextInt(6);
return new Money(UUID.randomUUID().toString(), type[i], currency[j], value[k], quality[l]);
}
}
@Data
@AllArgsConstructor
class Money {
private final String id, type, currency, value, quality;
}
Here we simply create the whole microservice in one class. The most important code is highlighted here. SUPER SIMPLE! Now you already have a microservice, which prints money and publishes it to the destination topic/pipeline “processor” 👏.
Implementation Processor
Implementation Consumer
Docker for Kafka and Zookeeper
Run these commands to create a network and run Kafka and Zookeeper in docker containers.
docker network create kafka
docker run -d --net=kafka --name=zookeeper -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:5.0.0
docker run -d --net=kafka --name=kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc/cp-kafka:5.0.0
If you can’t connect, add this line to /etc/hosts to ensure proper routing to container network “kafka”:
127.0.0.1 kafka
Start messaging platforms with the docker start command:
docker start zookeeper
docker start kafka
It’s a wrap!
Congratulations! You made it. Now just run your producer, processor and consumer and it should look something like this:
My example
Getting started
- Run docker/runKafka.sh
- Run docker/startMessagingPlatforms.sh
- Start producer, processor and consumer microservice (e.g. inside IntelliJ)
- Enjoy the log output 👨💻📋
Download the source code
The whole project is freely available on our Gitlab repository. Feel free to fix any mistakes and to comment here if you have any questions or feedback.
https://gitlab.com/47northlabs/public/spring-cloud-stream-money
