Reduce Redundant Producers from Partitioned Producer #PulsarSummit

Reduce redundant producers from partitioned producer Yuri Mizushima - Yahoo Japan Corporation ©2021 Yahoo Japan Corporation All rights reserved.

Agenda 1. Background of the issue 2. Solution 3. Benchmark 4. Conclusion ©2021 Yahoo Japan Corporation All rights reserved. 2

Agenda 1. Background of the issue 2. Solution 3. Benchmark 4. Conclusion ©2021 Yahoo Japan Corporation All rights reserved. 3

1. Background of the issue Apache Pulsar in Yahoo! JAPAN In Yahoo! JAPAN, Apache Pulsar is used in many use cases. • Notification of contents update • Job queuing • etc. ©2021 Yahoo Japan Corporation All rights reserved. 4

1. Background of the issue One of use cases, metrics/logs streaming pipeline Also, Pulsar is used in metrics and logs streaming pipeline. ©2021 Yahoo Japan Corporation All rights reserved. 5

1. Background of the issue One of use case, metrics/logs stream pipeline In this case, an unspecified number of producers connect to partitioned topics. It causes 1. The number of producers exceeds the limit 2. Redundant producers are created per instances ©2021 Yahoo Japan Corporation All rights reserved. 6

1. Background of the issue One of use case, metrics/logs stream pipeline In this case, an unspecified number of producers connect to partitioned topics. It causes 1. The number of producers exceeds the limit 2. Redundant producers are created per instances ©2021 Yahoo Japan Corporation All rights reserved. 7

1. Background of the issue The number of producers exceeds the limit Pulsar has a config maxProducersPerTopic. ”Ill-behaved" clients which increase producers infinitely can make the topic producer-full. To solve this issue, • Limit the number of producers/consumers that can connect per topic for each IP address (https://github.com/apache/pulsar/pull/10188) • Even if the number of producer is increased by a client, the influence is suppressed within only single IP address • I donʼt explain the detail in this session ©2021 Yahoo Japan Corporation All rights reserved. 8

• https://github.com/apache/pulsar/pull/10188

1. Background of the issue One of use case, metrics/logs stream pipeline In this case, an unspecified number of producers connect to partitioned topics. It causes 1. The number of producers exceeds the limit 2. Redundant producers are created per instances ©2021 Yahoo Japan Corporation All rights reserved. 9

1. Background of the issue Redundant producers are created per instances When a producer connects to the partitioned topic, sometimes it has redundant internal producers. 1. Relatively "low-rate" producers • The number of partitions needs to be increased according to total throughput • However, creating internal producers for all partitions is inefficient for producers whose throughput is small enough to be handled by a few partitions 2. SinglePartition routing mode Each producer uses only one partition • • other internal producers are redundant ©2021 Yahoo Japan Corporation All rights reserved. 10

1. Background of the issue Next... In this session, 1. The number of producers exceeds the limit 2. Redundant producers are created per instances ©2021 Yahoo Japan Corporation All rights reserved. 11

Agenda 1. Background of the issue 2. Solution 3. Benchmark 4. Conclusion ©2021 Yahoo Japan Corporation All rights reserved. 12

2. Solution Concepts Reduce the number of producers to use system resources (e.g. broker heap, number of TCP connections, client heap, etc.) more efficiently. ©2021 Yahoo Japan Corporation All rights reserved. 13

2. Solution Detail - limiting by number of internal producers When a partitioned producer connects to the topic, the client can randomly choose the limiting number of internal producers and internal partitions as well. ©2021 Yahoo Japan Corporation All rights reserved. 14

2. Solution Detail - lazy-loading First, at initialization step, a partitioned producer connects to only one of partitions for authn. and authz. instead of all partitions. When the internal producer is created, validate authn., and validate authz. at this topic. ©2021 Yahoo Japan Corporation All rights reserved. 15

2. Solution Detail - lazy-loading Second, at message sending step, partitions are chosen by message router. Each internal producer is created on the first time to be chosen by message router. ©2021 Yahoo Japan Corporation All rights reserved. 16

2. Solution Detail - partial round-robin Also, we add new custom routing mode PartialRoundRobinMessageRouter. This mode supports round-robin with limiting the number of partitions. ©2021 Yahoo Japan Corporation All rights reserved. 17

2. Solution Detail - partitioned producer stats Partitioned producer stats are accumulated for all partitions. Accumulating procedure supposes “all the partitions have the same number of producer”. ©2021 Yahoo Japan Corporation All rights reserved. 18

2. Solution Detail - partitioned producer stats We introduce producerStatsKey. Publisher stats with the same value for this property are accumulated as the same producer. ©2021 Yahoo Japan Corporation All rights reserved. 19

Agenda 1. Background of the issue 2. Solution 3. Benchmark 4. Conclusion ©2021 Yahoo Japan Corporation All rights reserved. 20

3. Benchmark Criterion • Broker heap usage, CPU percentage • Client heap usage, CPU percentage • Number of TCP connections between client and broker • Producer initialization time ©2021 Yahoo Japan Corporation All rights reserved. 21

3. Benchmark Assumption • Send any messages without batching • Send any message with PartialRoundRobinMessageRouter(with proposed) and RoundRobinPartition(with existing) • Create a partitioned topic with 30 partitions • Send 1024 bytes messages with 5000 rps • Existing • commit hash: 0b71b13 ©2021 Yahoo Japan Corporation All rights reserved. 22

3. Benchmark Procedure and Variables Procedure • 1. Run b broker servers 2. Create a producer limiting l partitions 3. Send messages for 10 minutes 4. (Check stats) Variables • • b=5, l=3 • b=5, l=5 • b=10, l=5 ©2021 Yahoo Japan Corporation All rights reserved. 23

3. Benchmark Environment • macOS 10.15.7 • 2.5GHz dual-core Intel Core i7 • 16 GB 2133 MHz LPDDR3 • AdoptOpenJDK-11.0.11+9 ©2021 Yahoo Japan Corporation All rights reserved. 24

3. Benchmark Result - b=5, l=3 Number of TCP connections between client and broker • • proposed: 3 • existing: 5 • proposed one is less than existing one Producer initialization time • • proposed: 463 [ms] • existing: 1389 [ms] ©2021 Yahoo Japan Corporation All rights reserved. 25

3. Benchmark Result - b=5, l=3 1.20GB 250MB proposed 1.20GB existing 250MB client heap broker heap proposed one is less than existing one ©2021 Yahoo Japan Corporation All rights reserved. 26

3. Benchmark Result - b=5, l=3 proposed 40% 10% 40% existing 10% client CPU broker CPU proposed one has a larger variance than existing one ©2021 Yahoo Japan Corporation All rights reserved. 27

3. Benchmark Result - b=5, l=5 Number of TCP connections between client and broker • • proposed: 5 • existing: 5 Producer initialization time • • proposed: 862 [ms] • existing: 1389 [ms] ©2021 Yahoo Japan Corporation All rights reserved. 28

3. Benchmark Result - b=5, l=5 1.20GB 250MB proposed 1.20GB existing 250MB client heap broker heap ©2021 Yahoo Japan Corporation All rights reserved. 29

3. Benchmark Result - b=5, l=5 40% proposed 10% 40% existing 10% client CPU broker CPU ©2021 Yahoo Japan Corporation All rights reserved. 30

3. Benchmark Result - b=10, l=5 Number of TCP connections between client and broker • • proposed: 4 • existing: 5 Producer initialization time • • proposed: 1070 [ms] • existing: 1147 [ms] ©2021 Yahoo Japan Corporation All rights reserved. 31

3. Benchmark Result - b=10, l=5 1.20GB 250MB proposed 1.20GB 250MB existing client heap broker heap ©2021 Yahoo Japan Corporation All rights reserved. 32

3. Benchmark Result - b=10, l=5 proposed 40% 10% 40% existing 10% client CPU broker CPU ©2021 Yahoo Japan Corporation All rights reserved. 33

3. Benchmark Consideration Client side • • Producer initialization time is faster than “existing” • Number of TCP connections is less than or equal to “existing” • Suppose a cluster has enough brokers. If we use the feature, number of TCP connections is less than or equal to “limit+1” • Heap usage is less than “existing” • Some cases of CPU percentage is stabler than “existing” ©2021 Yahoo Japan Corporation All rights reserved. 34

3. Benchmark Consideration Number of TCP connections Producer initialization time [ms] proposed existing proposed existing b=5, l=3 3 5 463 1389 b=5, l=5 5 5 862 1389 b=10, l=5 4 5 1070 1147 ©2021 Yahoo Japan Corporation All rights reserved. 35

3. Benchmark Consideration Broker side • • No significant effects to heap usage • CPU percentage per broker has a larger variance than “existing” • Maybe because topic load isnʼt distributed to all brokers • Number of running brokers (whose CPU/heap usages were increased) is less than "existing" • Number of running brokers depends on "active topics" where producers are actually connected ©2021 Yahoo Japan Corporation All rights reserved. 36

3. Benchmark Consideration Number of brokers whose system resources were increased proposed existing b=5, l=3 2 4 b=5, l=5 4 4 b=10, l=5 3 4 b=5, l=3 Number of topics which is loaded proposed existing broker0 0 8 broker1 1 0 broker2 2 7 broker3 0 7 broker4 0 8 ©2021 Yahoo Japan Corporation All rights reserved. 37

Agenda 1. Background of the issue 2. Solution 3. Benchmark 4. Conclusion ©2021 Yahoo Japan Corporation All rights reserved. 38

4. Conclusion Conclusion In conclusion, Implement producer lazy-loading and partial round-robin feature • • and fix partitioned producer stats collecting logic Check the performance by toy example and find the pros and cons • • The feature affects to the client resource and broker load • client resource: particularly, number of TCP connections is less than or equal to “existing” • broker load: particularly, CPU percentage per broker has a larger variance than “existing” ©2021 Yahoo Japan Corporation All rights reserved. 39

4. Conclusion Conclusion In conclusion, Future tasks • • Also implement the feature to other clients • e.g. C++, Go, etc. • Implement smart message router • e.g. considering current load Iʼm really excited to be involved in Apache Pulsar. ©2021 Yahoo Japan Corporation All rights reserved. 40

©2021 Yahoo Japan Corporation All rights reserved.