Want to dive even deeper?

Take the course Building Real-time Communication Applications Using Twilio by Michael Peacock and become an expert!
Building Real-time Communication Applications Using Twilio
by Michael Peacock

Check it out!
You're watching a preview of this video, click the button on the left to puchase the full version from Atomikos ExtremeTransactions Certification.

Real-World Akka Recipes

In the brave new world of actor programming conventional design patterns frequently are not applicable, as is witnessed by questions we get on the mailing list and at conferences. That is why we have collected a number of common solutions and best practices for solving typical problems you will encounter when building scalable and robust systems with Akka actors. In this session we will show you how to implement flow control, distributed workers, blocking resources, reliable messaging and more.

Published on
  • 5.938
  • 61
  • 0
  • 11
  • 1
  • Real World Akka Recipes Jamie Allen Björn Antonsson Patrik Nordwall
  • The Fallacy of Guaranteed Delivery Jamie Allen @jamie_allen
  • Guaranteed Delivery • From Enterprise Integration Patterns • Messaging system uses built-in store to persist • ACK everywhere – Producer to sender – Sender to receiver – Receiver to consumer @jamie_allen 3
  • Akka Guarantees • Not much, intentionally • At most once, with no reordering • Pick your poison: – At most once – At least once – Exactly once • You have to add it on top @jamie_allen 4
  • How Do I Do It? • Handle “at least” semantics on receiver to deal with duplicates – Idempotent behavior in receiver – Check message ID • Handle “at most” semantics on the sender via retries – ACK every time message is handled – Cancel repeated send @jamie_allen 5
  • Durable Mailboxes? Uh,  no. @jamie_allen 6
  • Durable Mailboxes • Doesn’t work with future-based message sending (ask, ?) • No guarantee is there that the message even got to the mailbox in distributed systems • Asking for guarantees in an uncertain world @jamie_allen 7
  • Event Sourcing? • Wonderful pattern for compiling a list of timeseries events • Separation of concerns from actor mailboxes • Still lots of things that can go wrong – Disk writing – Replication consistency – Network partitions – Application latency @jamie_allen 8
  • External Durable Message Queue • You still have to ACK • No certainty the message you needed even got this far • Additional dependencies in your architecture @jamie_allen 9
  • Guaranteed Delivery Doesn’t Exist • We don’t know what we don’t know • Increased effort • Increased complexity • Increased latency • No guarantees of consistency • Doesn’t guarantee ordering @jamie_allen 10
  • So What Do We Do? • This falls outside of actor supervision; nothing the actors know about has gone wrong • Listen to Roland Kuhn: “Recovery ... should ideally be automatic in order to restore normal service as quickly as possible.” @jamie_allen 11
  • Sentinels @jamie_allen 12
  • Sentinels • Supervisors handle failure BELOW them. Sentinels handle failure ABOVE. • Responsible for querying a “source of truth” and getting latest state • Sends data to supervisor, who resolves differences in which instances of actors should exist versus those that do • Supervisor forwards data to instances that should exist for them to resolve their internal state @jamie_allen 13
  • Missed Events Add  Customer  3 Delete  Customer  1 Update  Customer  2 Customer  1 @jamie_allen Customer   Supervisor Customer  2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer  1 @jamie_allen Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer  1 @jamie_allen Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 X Customer  1 @jamie_allen Customer   Supervisor Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer   Sentinel Customer  2 Customer   2 @jamie_allen
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer   Sentinel Customer   2 @jamie_allen Customer   3
  • Sentinels • Localize them for each kind of data that must be synchronized in your supervisor hierarchy • Do not create one big one and try to resolve the entire tree at once @jamie_allen 20
  • Drawbacks • Doesn’t work well with localized event sourcing - time series can be lost • Does introduce additional complexity and tunable latency over applications with no guarantees • Pattern only works when there is a queryable source of truth @jamie_allen 21
  • Inconsistent Views? • Using Sentinels at multiple levels of a supervisory hierarchy can lead to temporarily inconsistent views when child actors are resolved before parents on delete (no atomicity) • But is this necessarily bad? @jamie_allen 22
  • Sentinels in Hierarchy S CS C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS S DS D1 @jamie_allen S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS S DS D1 @jamie_allen S D2
  • Sentinels in Hierarchy CS C1 @jamie_allen S
  • A Huge Win • Your system is resilient to external failures • You can tune sentinel update frequency to meet changing requirements • Your system is considerably less complex than attempting to guarantee no message loss @jamie_allen 29
  • Flow Control Björn Antonsson @bantonsson
  • Pure Push Applications • Often the first Actor application you write – Once you start telling and stop asking • Easy to implement and reason about • Fits nicely with short lived jobs that come at a fixed rate @bantonsson 31
  • @bantonsson 32
  • Why do you need anything else? • Produce jobs faster than you can finish them • Jobs are expensive compute/memory wise • External resources impose limits • Unpredictable job patterns @bantonsson 33
  • What can you do instead? • Push with rate limiting – A fixed number of jobs per time unit are pushed • Push with acknowledgment – A fixed number of jobs can be in progress. – New jobs are pushed after old jobs finish • Pull – Jobs are pulled from the master at the rate that they are completed @bantonsson 34
  • Push with rate limiting • A timer sends the master ticks at fixed intervals • When a tick arrives, the master fills up its token count • If a job arrives and there are no tokens, it gets queued • When the master has tokens, it pulls jobs off the queue and pushes them @bantonsson 35
  • @bantonsson 36
  • Push with acknowledgement • The master push a fixed number of jobs before waiting for an acknowledgement • If a job arrives and the master can't push, it gets queued • To keep workers busy, push more than one job per worker – You can use a high water mark to stop and a low water mark to start pushing @bantonsson 37
  • @bantonsson 38
  • Pull • The master actor queues incoming jobs • Worker actors ask the master for a job and receives jobs when available • The workers don't need to do active polling • Can lead to lag if jobs are small compared to the time it takes to get a new one – Use batching to counteract lag @bantonsson 39
  • @bantonsson 40
  • References • Push with rate limiting – Kaspar Fischer http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2 • Pull – Derek Wyatt http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-withakka-2 – Michael Pollmeier http://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/ @bantonsson 41
  • Distributed Workers Patrik Nordwall @patriknw
  • 43
  • Goal • elastic addition/removal of front end nodes • elastic addition/removal of workers • thousands of workers • jobs should not be lost 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • DistributedPubSubMediator 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 56
  • 56
  • 54
  • 52
  • 49

Comments

Be the first one to add a comment

Thumbnail for: Busy Java Developer’s Guide to Three REST API Frameworks

Busy Java Developer’s Guide to Three REST API Frameworks

Ted Neward

subscription
Thumbnail for: Corporate startup and Scala

Corporate startup and Scala

Filip Rogaczewski

subscription
Thumbnail for: Co było pierwsze: kod czy architektura?

Co było pierwsze: kod czy architektura?

Slawomir Sobotka

subscription
Thumbnail for: Preventing Analytics and Ads from polluting your codebase (no slides)

Preventing Analytics and Ads from polluting your codebase (no slides)

Savvas Dalkitsis

subscription
Thumbnail for: From Docker To Kubernetes: A Developer's Guide To Containers

From Docker To Kubernetes: A Developer's Guide To Containers

Marek Grabowski

subscription
Thumbnail for: Ścisły przewodnik po aspektach miękkich - część II: Jesteś ekspertem i co dalej?

Ścisły przewodnik po aspektach miękkich - część II: Jesteś ekspertem i co dalej?

Slawomir Sobotka

free
Thumbnail for: Designing software with security in mind?

Designing software with security in mind?

Daniel Deogun

subscription
Thumbnail for: OAuth2 for native apps and beyond

OAuth2 for native apps and beyond

Erik Jan de Wit

subscription
Thumbnail for: Using JavaScript/HTML5 Rich Clients with Java EE 7

Using JavaScript/HTML5 Rich Clients with Java EE 7

Reza Rahman

subscription
Thumbnail for: Why I love Logstash and you should too

Why I love Logstash and you should too

João Duarte

subscription
Thumbnail for: Java Bytecode Explained

Java Bytecode Explained

OLEG ŠELAJEV

subscription
Thumbnail for: Agile Project Management Anti-Patterns

Agile Project Management Anti-Patterns

Kasia Mrowca

subscription
Thumbnail for: Applications secure by default

Applications secure by default

Sławomir Jasek

subscription
Thumbnail for: From spaghetti with no src/test to green CI, good coverage and well-sleeping developers

From spaghetti with no src/test to green CI, good coverage and well-sleeping developers

Michał Matłoka , Jacek Kunicki

subscription
Thumbnail for: Failing Continuous Delivery

Failing Continuous Delivery

Daniel Sawano , Daniel Deogun

subscription
Thumbnail for: Tips and tricks for clean relational db schemas (slides only)

Tips and tricks for clean relational db schemas (slides only)

Clément Delafargue

subscription
Thumbnail for: Production Lessons (slides only)

Production Lessons (slides only)

Greg Young

subscription

The Spring BOF (no slides)

Juergen Hoeller , Josh Long

subscription

BigData on Azure for architects – Machine Learning, HDInsight (Hadoop), Event Hub – what to use when (and – how) (slides only)

Tomasz Kopacz

subscription
Thumbnail for: "Bootiful" Applications with Spring Boot

"Bootiful" Applications with Spring Boot

Josh Long

subscription
Thumbnail for: 5-10-15 years with Java - from junior to master and back again (slides only)

5-10-15 years with Java - from junior to master and back again (slides only)

Wojciech Seliga

subscription
Thumbnail for: Everything Works In Java EE?--Then Try Microservices :-) (slides only)

Everything Works In Java EE?--Then Try Microservices :-) (slides only)

Adam Bien

subscription
Thumbnail for: Supler: complex web forms, not so complex

Supler: complex web forms, not so complex

Adam Warski , Tomasz Szymanski

subscription
Thumbnail for: Functional Data Storage (slides only)

Functional Data Storage (slides only)

Greg Young

subscription
Thumbnail for: The Go Language (slides only)

The Go Language (slides only)

Brad Fitzpatrick

subscription
Thumbnail for: The Silver Bullet Syndrome (slides only)

The Silver Bullet Syndrome (slides only)

Hadi Hariri

subscription
Thumbnail for: What's Coming in Java EE 8

What's Coming in Java EE 8

Reza Rahman

subscription
Thumbnail for: From API to protocol (slides only)

From API to protocol (slides only)

Quentin ADAM

subscription
Thumbnail for: Babun - a Windows shell you will love, finally! (slides only)

Babun - a Windows shell you will love, finally! (slides only)

Tom Bujok , Lukasz Pielak

subscription
Thumbnail for: IDE Wizard? CLI Command? Get both for the price of one! (slides only)

IDE Wizard? CLI Command? Get both for the price of one! (slides only)

Koen Aers

subscription
Thumbnail for: Event Sourcing & Functional Programming - a pair made in heaven

Event Sourcing & Functional Programming - a pair made in heaven

Paweł Szulc

subscription
Thumbnail for: Everybody lies

Everybody lies

Tomasz Kowalczewski

free
Thumbnail for: Scala and Clojure: Playing well together

Scala and Clojure: Playing well together

David Pollak

subscription
Thumbnail for: Groovier BDD with Spock (slides only)

Groovier BDD with Spock (slides only)

Michał Kordas

subscription
Thumbnail for: Clean JavaScript code - only dream or reality (slides only)

Clean JavaScript code - only dream or reality (slides only)

Sebastian Łaciak

subscription
Thumbnail for: Wykorzystanie podejścia Domain-Driven Design w systemach legacy (slides only)

Wykorzystanie podejścia Domain-Driven Design w systemach legacy (slides only)

Piotr Wyczesany

subscription
Thumbnail for: Case Study: Agile @gov.pl (slides only)

Case Study: Agile @gov.pl (slides only)

Matt Harasymczuk

subscription
Thumbnail for: Functional patterns for scala beginners (slides only)

Functional patterns for scala beginners (slides only)

Clément Delafargue

subscription
Thumbnail for: Vagrant up your enviroment (slides only)

Vagrant up your enviroment (slides only)

Yann Larrivee

subscription
Thumbnail for: Running Micro-Services with Spring Boot in Kubernetes (slides only)

Running Micro-Services with Spring Boot in Kubernetes (slides only)

Ray Tsang

subscription
Thumbnail for: Scalable Is Awesome, Literally! (slides only)

Scalable Is Awesome, Literally! (slides only)

Garrett Smith

subscription
Thumbnail for: Apache Spark - when things go wrong (slides only)

Apache Spark - when things go wrong (slides only)

Paweł Szulc

subscription
Thumbnail for: Functional Programming: Technical Reasons to Adapt (screen only)

Functional Programming: Technical Reasons to Adapt (screen only)

Venkat Subramaniam

subscription
Thumbnail for: PubSub++ - few tips that make your life with kafka easier. (slides only)

PubSub++ - few tips that make your life with kafka easier. (slides only)

Krzysztof Debski

subscription
Thumbnail for: Effective SCRUM in distributed teams (slides only)

Effective SCRUM in distributed teams (slides only)

Rafał Udziela

subscription
Thumbnail for: React.js: Super-fast Single Page Web Applications (slides only)

React.js: Super-fast Single Page Web Applications (slides only)

Pratik Patel

subscription
Thumbnail for: The Smartwatch Revolution begins: Developing for Android Wear (slides only)

The Smartwatch Revolution begins: Developing for Android Wear (slides only)

Pratik Patel

subscription
Thumbnail for: Apache Cassandra 101 (slides only)

Apache Cassandra 101 (slides only)

Christopher Batey

subscription
Thumbnail for: Distributed algorithms for Big Data (slides only)

Distributed algorithms for Big Data (slides only)

DuyHai DOAN

subscription
Thumbnail for: What's new in Spring Data? (slides only)

What's new in Spring Data? (slides only)

Thomas Darimont

subscription
Thumbnail for: User Acceptance Testing - Looking for the Holly Grail (slides only)

User Acceptance Testing - Looking for the Holly Grail (slides only)

Boguslaw Osuch

subscription
Thumbnail for: Is your profiler speaking the same language as you? (slides only)

Is your profiler speaking the same language as you? (slides only)

Simon Maple

subscription
Thumbnail for: Microservices and Conversion Hunting - How to build software architectures for changeableness (slides only)

Microservices and Conversion Hunting - How to build software architectures for changeableness (slides only)

Bernd Zuther

subscription
Thumbnail for: JRebel under the covers - how is it even possible? (slides only)

JRebel under the covers - how is it even possible? (slides only)

Simon Maple

subscription
Thumbnail for: Microservices - enough with theory, let's code some (slides only)

Microservices - enough with theory, let's code some (slides only)

Marcin Grzejszczak , Tomasz Szymanski

subscription
Thumbnail for: Core Software Design Principles for Programmers (slides only)

Core Software Design Principles for Programmers (slides only)

Venkat Subramaniam

subscription
Thumbnail for: A Gentle and useful introduction to Reactive Extensions (slides only)

A Gentle and useful introduction to Reactive Extensions (slides only)

Hadi Hariri

subscription
Thumbnail for: Caching reboot: javax.cache & Ehcache 3 (slides only)

Caching reboot: javax.cache & Ehcache 3 (slides only)

Louis Jacomet

subscription
Thumbnail for: Lazy Evaluations (screen only)

Lazy Evaluations (screen only)

Venkat Subramaniam

subscription
Thumbnail for: Make sense of your (BIG) data! (slides only)

Make sense of your (BIG) data! (slides only)

David Pilato

subscription
Thumbnail for: The end of server management : hosting have to become a commodity (slides only)

The end of server management : hosting have to become a commodity (slides only)

Quentin ADAM

subscription
Thumbnail for: Refactoring to Functional (slides only)

Refactoring to Functional (slides only)

Hadi Hariri

subscription
Thumbnail for: HTTP/2 : why upgrading the web? (slides only)

HTTP/2 : why upgrading the web? (slides only)

Quentin ADAM

subscription
Thumbnail for: Hystrix – managing failures in distributed systems (slides only)

Hystrix – managing failures in distributed systems (slides only)

Tomasz Nurkiewicz

subscription
Thumbnail for: Building systems that are #neverdone

Building systems that are #neverdone

James Lewis

subscription
Thumbnail for: Need for Async: In hot pursuit of internet-scale app architectures

Need for Async: In hot pursuit of internet-scale app architectures

Konrad Malawski

subscription
Thumbnail for: Level up your dev skills with static analysis

Level up your dev skills with static analysis

David Lindsay

subscription
Thumbnail for: Modularity in post microservice world

Modularity in post microservice world

Michal Gruca

subscription
Thumbnail for: Modern Java Component Design with Spring 4.2

Modern Java Component Design with Spring 4.2

Juergen Hoeller

subscription
Thumbnail for: The Creative Networker

The Creative Networker

Jurgen Appelo

subscription
Thumbnail for: Technical leadership – from an expert to a leader

Technical leadership – from an expert to a leader

Mariusz Sieraczkiewicz

subscription
Thumbnail for: JCP, Adopt-a-JSR & You

JCP, Adopt-a-JSR & You

Reza Rahman

subscription
Thumbnail for: Lessons learned from scaling software

Lessons learned from scaling software

Bartek Nowakowski

subscription
Thumbnail for: Building fault tolerant microservices

Building fault tolerant microservices

Christopher Batey

subscription
Thumbnail for: WebVR - democracy in Virtual Reality

WebVR - democracy in Virtual Reality

Marcin Lichwala

subscription
Thumbnail for: High Performance JavaScript Web Apps

High Performance JavaScript Web Apps

Pratik Patel

subscription
Thumbnail for: Social Coding - tools and techniques

Social Coding - tools and techniques

Maciej Pleśnar

subscription
Thumbnail for: Why software developers should care about deployment and monitoring

Why software developers should care about deployment and monitoring

Michał Kosmulski

subscription
Thumbnail for: Building an Asynchronous Reactive NoSQL SDK with RxJava

Building an Asynchronous Reactive NoSQL SDK with RxJava

Simon Baslé

subscription
Thumbnail for: OnConnectionLost: The life of an offline web application

OnConnectionLost: The life of an offline web application

Stefanie Grewenig , Johannes Thönes

subscription
Thumbnail for: Deploying Microservice Architectures with Spring Cloud on Cloud Foundry

Deploying Microservice Architectures with Spring Cloud on Cloud Foundry

Pieter Humphrey

subscription
Thumbnail for: Agile Development Meets Connected Devices - Lessons Learned

Agile Development Meets Connected Devices - Lessons Learned

subscription
Thumbnail for: Ansible to rule them all – o provisioningu i deploymencie aplikacji Java

Ansible to rule them all – o provisioningu i deploymencie aplikacji Java

Wojciech Podgorski , Rafal Piotrowski

subscription
Thumbnail for: You're an Architect...Now What?

You're an Architect...Now What?

Nathaniel Schutta

subscription
Thumbnail for: Deep dive into Reactive Java

Deep dive into Reactive Java

Tomasz Kowalczewski

subscription
Thumbnail for: Nashorn, what is the whole buzz about

Nashorn, what is the whole buzz about

Michal Gruca

subscription
Thumbnail for: Things about microservices you wish you never knew

Things about microservices you wish you never knew

Marek Ko-w

subscription
Thumbnail for: "Bootiful" Microservices with Spring Cloud

"Bootiful" Microservices with Spring Cloud

Josh Long

subscription
Thumbnail for: Refactoring meets big money

Refactoring meets big money

Michal Gruca

subscription
Thumbnail for: Security Platform as a Service with Docker and Weave

Security Platform as a Service with Docker and Weave

David Pollak

subscription
Thumbnail for: Clean Architecture - how to improve your system architecture

Clean Architecture - how to improve your system architecture

Andrzej Bednarz

subscription
Thumbnail for: Principles Of Microservices

Principles Of Microservices

Sam Newman

subscription
Thumbnail for: Painfree Object-Document Mapping for MongoDB

Painfree Object-Document Mapping for MongoDB

Philipp Krenn

subscription
Thumbnail for: Reviewing Architectures

Reviewing Architectures

Nathaniel Schutta

subscription
Thumbnail for: Unlocking the magic of monads in Java 8

Unlocking the magic of monads in Java 8

OLEG ŠELAJEV

subscription
Thumbnail for: Kill the mutants, test your tests

Kill the mutants, test your tests

Roy van Rijn

subscription
Thumbnail for: Vaadin Designer, the visual design tool for modern web apps

Vaadin Designer, the visual design tool for modern web apps

Maciej Przepióra

subscription
Thumbnail for: My running shoes - Continuous Delivery

My running shoes - Continuous Delivery

Grzegorz Krumpholz

free
Thumbnail for: Get Past the Syntax, The Real Scare is in the Semantics

Get Past the Syntax, The Real Scare is in the Semantics

Venkat Subramaniam

free
Thumbnail for: Coding Culture

Coding Culture

Sven Peters

free
Thumbnail for: Software architecture as code

Software architecture as code

Simon Brown

subscription
Thumbnail for: Flavors of Concurrency in Java

Flavors of Concurrency in Java

OLEG ŠELAJEV

free
Thumbnail for: Modern Web Architecture

Modern Web Architecture

Ted Neward

subscription
Thumbnail for: 10 NoSQL databases you have to know

10 NoSQL databases you have to know

Tom Bujok

subscription
Thumbnail for: What's New in WildFly 9

What's New in WildFly 9

Tomasz Adamski

subscription