Want to dive even deeper?

Take the course Building Real-time Communication Applications Using Twilio by Michael Peacock and become an expert!
Building Real-time Communication Applications Using Twilio
by Michael Peacock

Check it out!
You're watching a preview of this video, click the button on the left to puchase the full version from JavaPolis'06.

Real-World Akka Recipes

In the brave new world of actor programming conventional design patterns frequently are not applicable, as is witnessed by questions we get on the mailing list and at conferences. That is why we have collected a number of common solutions and best practices for solving typical problems you will encounter when building scalable and robust systems with Akka actors. In this session we will show you how to implement flow control, distributed workers, blocking resources, reliable messaging and more.

Published on
  • 5.717
  • 58
  • 0
  • 10
  • 1
  • Real World Akka Recipes Jamie Allen Björn Antonsson Patrik Nordwall
  • The Fallacy of Guaranteed Delivery Jamie Allen @jamie_allen
  • Guaranteed Delivery • From Enterprise Integration Patterns • Messaging system uses built-in store to persist • ACK everywhere – Producer to sender – Sender to receiver – Receiver to consumer @jamie_allen 3
  • Akka Guarantees • Not much, intentionally • At most once, with no reordering • Pick your poison: – At most once – At least once – Exactly once • You have to add it on top @jamie_allen 4
  • How Do I Do It? • Handle “at least” semantics on receiver to deal with duplicates – Idempotent behavior in receiver – Check message ID • Handle “at most” semantics on the sender via retries – ACK every time message is handled – Cancel repeated send @jamie_allen 5
  • Durable Mailboxes? Uh,  no. @jamie_allen 6
  • Durable Mailboxes • Doesn’t work with future-based message sending (ask, ?) • No guarantee is there that the message even got to the mailbox in distributed systems • Asking for guarantees in an uncertain world @jamie_allen 7
  • Event Sourcing? • Wonderful pattern for compiling a list of timeseries events • Separation of concerns from actor mailboxes • Still lots of things that can go wrong – Disk writing – Replication consistency – Network partitions – Application latency @jamie_allen 8
  • External Durable Message Queue • You still have to ACK • No certainty the message you needed even got this far • Additional dependencies in your architecture @jamie_allen 9
  • Guaranteed Delivery Doesn’t Exist • We don’t know what we don’t know • Increased effort • Increased complexity • Increased latency • No guarantees of consistency • Doesn’t guarantee ordering @jamie_allen 10
  • So What Do We Do? • This falls outside of actor supervision; nothing the actors know about has gone wrong • Listen to Roland Kuhn: “Recovery ... should ideally be automatic in order to restore normal service as quickly as possible.” @jamie_allen 11
  • Sentinels @jamie_allen 12
  • Sentinels • Supervisors handle failure BELOW them. Sentinels handle failure ABOVE. • Responsible for querying a “source of truth” and getting latest state • Sends data to supervisor, who resolves differences in which instances of actors should exist versus those that do • Supervisor forwards data to instances that should exist for them to resolve their internal state @jamie_allen 13
  • Missed Events Add  Customer  3 Delete  Customer  1 Update  Customer  2 Customer  1 @jamie_allen Customer   Supervisor Customer  2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer  1 @jamie_allen Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer  1 @jamie_allen Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 X Customer  1 @jamie_allen Customer   Supervisor Customer   Sentinel Customer   2
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer   Sentinel Customer  2 Customer   2 @jamie_allen
  • Sentinels Customer  2 Customer  3 Customer   Supervisor Customer   Sentinel Customer   2 @jamie_allen Customer   3
  • Sentinels • Localize them for each kind of data that must be synchronized in your supervisor hierarchy • Do not create one big one and try to resolve the entire tree at once @jamie_allen 20
  • Drawbacks • Doesn’t work well with localized event sourcing - time series can be lost • Does introduce additional complexity and tunable latency over applications with no guarantees • Pattern only works when there is a queryable source of truth @jamie_allen 21
  • Inconsistent Views? • Using Sentinels at multiple levels of a supervisory hierarchy can lead to temporarily inconsistent views when child actors are resolved before parents on delete (no atomicity) • But is this necessarily bad? @jamie_allen 22
  • Sentinels in Hierarchy S CS C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS A1 @jamie_allen A2 S A2 DS D1 S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS S DS D1 @jamie_allen S D2
  • Sentinels in Hierarchy S CS X C2 C1 AS S DS D1 @jamie_allen S D2
  • Sentinels in Hierarchy CS C1 @jamie_allen S
  • A Huge Win • Your system is resilient to external failures • You can tune sentinel update frequency to meet changing requirements • Your system is considerably less complex than attempting to guarantee no message loss @jamie_allen 29
  • Flow Control Björn Antonsson @bantonsson
  • Pure Push Applications • Often the first Actor application you write – Once you start telling and stop asking • Easy to implement and reason about • Fits nicely with short lived jobs that come at a fixed rate @bantonsson 31
  • @bantonsson 32
  • Why do you need anything else? • Produce jobs faster than you can finish them • Jobs are expensive compute/memory wise • External resources impose limits • Unpredictable job patterns @bantonsson 33
  • What can you do instead? • Push with rate limiting – A fixed number of jobs per time unit are pushed • Push with acknowledgment – A fixed number of jobs can be in progress. – New jobs are pushed after old jobs finish • Pull – Jobs are pulled from the master at the rate that they are completed @bantonsson 34
  • Push with rate limiting • A timer sends the master ticks at fixed intervals • When a tick arrives, the master fills up its token count • If a job arrives and there are no tokens, it gets queued • When the master has tokens, it pulls jobs off the queue and pushes them @bantonsson 35
  • @bantonsson 36
  • Push with acknowledgement • The master push a fixed number of jobs before waiting for an acknowledgement • If a job arrives and the master can't push, it gets queued • To keep workers busy, push more than one job per worker – You can use a high water mark to stop and a low water mark to start pushing @bantonsson 37
  • @bantonsson 38
  • Pull • The master actor queues incoming jobs • Worker actors ask the master for a job and receives jobs when available • The workers don't need to do active polling • Can lead to lag if jobs are small compared to the time it takes to get a new one – Use batching to counteract lag @bantonsson 39
  • @bantonsson 40
  • References • Push with rate limiting – Kaspar Fischer http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2 • Pull – Derek Wyatt http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-withakka-2 – Michael Pollmeier http://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/ @bantonsson 41
  • Distributed Workers Patrik Nordwall @patriknw
  • 43
  • Goal • elastic addition/removal of front end nodes • elastic addition/removal of workers • thousands of workers • jobs should not be lost 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • DistributedPubSubMediator 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 56
  • 56
  • 54
  • 52
  • 49

Comments

Be the first one to add a comment

Thumbnail for: One Framework, No Dependencies: Using Frameworks to Avoid Common Pitfalls of App Development

One Framework, No Dependencies: Using Frameworks to Avoid Common Pitfalls of App Development

Lee Boonstra

free
Thumbnail for: Java EE Game Changers

Java EE Game Changers

David Blevins

free
Thumbnail for: Five Ways to Not Suck at Being a Java Freelancer

Five Ways to Not Suck at Being a Java Freelancer

Roberto Cortez

free
Thumbnail for: Express yourself with Java 8

Express yourself with Java 8

Benjamin Weber

free
Thumbnail for: Never Mind the Devoxx

Never Mind the Devoxx

David Blevins

free
Thumbnail for: Building the UKs biggest IoT network

Building the UKs biggest IoT network

Romanas Sonkinas

free
Thumbnail for: The First IoT JSR: Units of Measurement

The First IoT JSR: Units of Measurement

Leonardo Lima , Chris Senior

free
Thumbnail for: Devoxx4Kids

Devoxx4Kids

Dan Hardiker

free
Thumbnail for: Ceylon from here to infinity: the big picture and what's coming

Ceylon from here to infinity: the big picture and what's coming

Stephane Epardaud

free
Thumbnail for: Predicting the Future with Machine Learning

Predicting the Future with Machine Learning

Amy Nicholson

free
Thumbnail for: Microservices Made Easy with Spring Cloud and Netflix OSS

Microservices Made Easy with Spring Cloud and Netflix OSS

David Syer

free
Thumbnail for: WildFly 9

WildFly 9

Kabir Khan

free
Thumbnail for: CDI 2.0: what is in the work?

CDI 2.0: what is in the work?

Antoine Sabot-Durand , José Paumard

free
Thumbnail for: Diversity Rules!

Diversity Rules!

Amali de Alwis

free
Thumbnail for: Top 10 Real-life WebSocket Use Cases and Experiences

Top 10 Real-life WebSocket Use Cases and Experiences

Richard Cullen

free
Thumbnail for: Microservices Bootcamp

Microservices Bootcamp

Josh Long

free
Thumbnail for: Flyway - Database Migration made easy

Flyway - Database Migration made easy

Axel Fontaine

free
Thumbnail for: How to Rock your Web Apps with Spring and Vaadin

How to Rock your Web Apps with Spring and Vaadin

Sami Ekblad

free
Thumbnail for: Software Architecture as Code

Software Architecture as Code

Simon Brown

free
Thumbnail for: Combining R with Java for Data Analysis

Combining R with Java for Data Analysis

Ryan Cuprak , Elsa Cuprak

free
Thumbnail for: Let’s Get to the Rapids: Java 8 Stream Perfomance

Let’s Get to the Rapids: Java 8 Stream Perfomance

Maurice Naftalin

free
Thumbnail for: Level Up your Dev Skills with Static Analysis

Level Up your Dev Skills with Static Analysis

David Lindsay

free
Thumbnail for: Akka Streams with Scala and Java

Akka Streams with Scala and Java

Jamie Allen

free
Thumbnail for: Functional patterns for scala beginners

Functional patterns for scala beginners

Clément Delafargue

free
Thumbnail for: The 10 Golden Rules of Mobile UX

The 10 Golden Rules of Mobile UX

Emilia Ciardi

free
Thumbnail for: HTTP/2 comes to Java. What Servlet 4.0 means to You

HTTP/2 comes to Java. What Servlet 4.0 means to You

David Delabassee

free
Thumbnail for: Is your profiler speaking the same language as you?

Is your profiler speaking the same language as you?

Simon Maple

free
Thumbnail for: Programming at any Scale

Programming at any Scale

Andrew Harmel-Law

free
Thumbnail for: Building Cloud Powered Android Apps with Azure

Building Cloud Powered Android Apps with Azure

Chris Risner

free
Thumbnail for: 5 reasons why developers should pay attention to robotics

5 reasons why developers should pay attention to robotics

Nicolas Rigaud

free
Thumbnail for: From 0 to 60 million users: Scaling for speed and high availability

From 0 to 60 million users: Scaling for speed and high availability

Aviran Mordo

free
Thumbnail for: In-Memory Computing - Distributed Systems

In-Memory Computing - Distributed Systems

Christoph Engelbert

free
Thumbnail for: Android Testing: A New Hope

Android Testing: A New Hope

Zan Markan , Nicholas Jackson

free
Thumbnail for: Java EE changes design pattern implementation

Java EE changes design pattern implementation

Alex Theedom

free
Thumbnail for: Making Enterprise Java Architecture Sustainable

Making Enterprise Java Architecture Sustainable

Markus Eisele

free
Thumbnail for: Ludicrously Fast Builds with Buck

Ludicrously Fast Builds with Buck

Simon Stewart

free
Thumbnail for: Java is dead, long live Ceylon, Kotlin,…

Java is dead, long live Ceylon, Kotlin,…

Russel Winder

free
Thumbnail for: Java 9: Make Way for Modules!

Java 9: Make Way for Modules!

Mark Reinhold

free
Thumbnail for: Comparing different concurrency models on the JVM

Comparing different concurrency models on the JVM

Mario Fusco

free
Thumbnail for: 50 minutes to develop a full Java EE application with Forge? Is that all?

50 minutes to develop a full Java EE application with Forge? Is that all?

Antonio Goncalves

free
Thumbnail for: Making Events for Local Dev Communities

Making Events for Local Dev Communities

Celestino Bellone , Federico Yankelevich

free
Thumbnail for: Refactor your Java EE application using Microservices and Containers

Refactor your Java EE application using Microservices and Containers

Arun Gupta

free
Thumbnail for: Welcome Keynote

Welcome Keynote

Mark Hazell

free
Thumbnail for: Developing iOS applications with Java and Eclipse

Developing iOS applications with Java and Eclipse

Grant Ronald

free
Thumbnail for: Java EE 8, a snapshot overview

Java EE 8, a snapshot overview

David Delabassee

free
Thumbnail for: Finally, Security API JSR 375

Finally, Security API JSR 375

Alex Kosowski

free
Thumbnail for: Decoding the air around you with Java and $7 hardware

Decoding the air around you with Java and $7 hardware

Bert Jan Schrijver

free
Thumbnail for: The Dark arts of building your own framework

The Dark arts of building your own framework

Andrey Glaschenko , Konstantin Krivopustov

free
Thumbnail for: Practical Continuous Deployment

Practical Continuous Deployment

Steve Smith

free
Thumbnail for: The Seven Deadly Sins of Microservices

The Seven Deadly Sins of Microservices

Daniel Bryant

free
Thumbnail for: Nuts and Bolts of WebSocket

Nuts and Bolts of WebSocket

Arun Gupta

free
Thumbnail for: Level up your Android Build

Level up your Android Build

Volker Leck

free
Thumbnail for: Powering real-time web apps with Hazelcast

Powering real-time web apps with Hazelcast

Andrejs Jermakovics

free
Thumbnail for: What is your build telling you about your application structure?

What is your build telling you about your application structure?

Abraham Marin-Perez

free
Thumbnail for: Practical Challenges of moving from CI to CD using Maven

Practical Challenges of moving from CI to CD using Maven

Jatin Bhadra

free
Thumbnail for: Design Patterns in the 21st Century

Design Patterns in the 21st Century

Samir Talwar

free
Thumbnail for: All Change! How the new economics of Cloud will make you think differently about Java

All Change! How the new economics of Cloud will make you think differently about Java

Steve Poole

free
Thumbnail for: Destination: Cloud. Deploying Applications To The Cloud With Docker

Destination: Cloud. Deploying Applications To The Cloud With Docker

Ryan Baxter

free
Thumbnail for: Not my department?! How our privacy is broken and what you can do about it.

Not my department?! How our privacy is broken and what you can do about it.

Anna Biselli

free
Thumbnail for: goto java;

goto java;

free
Thumbnail for: The Hidden Gems of PDF

The Hidden Gems of PDF

Michaël Demey

free
Thumbnail for: Accessibility and how to get the most from your screenreader

Accessibility and how to get the most from your screenreader

Edd Sowden

free
Thumbnail for: Building securely with agile

Building securely with agile

Michael Brunton-spall

free
Thumbnail for: STOP BREAKING MY STUFF

STOP BREAKING MY STUFF

Katharine Fitzpatrick

free
Thumbnail for: Java Generics: Past, Present and Future

Java Generics: Past, Present and Future

Richard Warburton , Raoul-Gabriel Urma

free
Thumbnail for: Keynote: The Extraordinary Team of Developers

Keynote: The Extraordinary Team of Developers

Trisha Gee , Daniel Bryant

free
Thumbnail for: Java 8 in Anger

Java 8 in Anger

Trisha Gee

free
Thumbnail for: Miniboxing: Fast Generics for Primitive Types

Miniboxing: Fast Generics for Primitive Types

Vlad Ureche

free
Thumbnail for: Bringing IBM Watson to the Cloud

Bringing IBM Watson to the Cloud

Dale Lane

free
Thumbnail for: The end of server management : hosting have to become a commodity

The end of server management : hosting have to become a commodity

Quentin ADAM

free
Thumbnail for: 3 Keys to Rapid Deployment Success

3 Keys to Rapid Deployment Success

Sven Erik Knop

free
Thumbnail for: Experimenting with ServiceWorker

Experimenting with ServiceWorker

Sandro Paganotti

free
Thumbnail for: "Catch me if you can" - Java on wearables

"Catch me if you can" - Java on wearables

Gerrit Grunwald

free
Thumbnail for: The Google Cloud Plaform - Beyond simple use cases

The Google Cloud Plaform - Beyond simple use cases

David Gageot

free
Thumbnail for: The Smartwatch Revolution begins: Developing for Android Wear

The Smartwatch Revolution begins: Developing for Android Wear

Pratik Patel

free
Thumbnail for: SQL’s back: querying document databases the old fashioned way

SQL’s back: querying document databases the old fashioned way

Matthew Revell

free
Thumbnail for: JCP, Adopt-a-JSR & You

JCP, Adopt-a-JSR & You

Heather VanCura

free
Thumbnail for: Pushing The Limits With Java Enterprise Cloud Development

Pushing The Limits With Java Enterprise Cloud Development

Simon Ritter

free
Thumbnail for: Performance Tuning, Top 10 Your Doing it Wrong!

Performance Tuning, Top 10 Your Doing it Wrong!

Kirk Pepperdine

free
Thumbnail for: What Were You Doing In 1995? A 20 Year Retrospective of Java

What Were You Doing In 1995? A 20 Year Retrospective of Java

Simon Ritter , Steve Elliott

free
Thumbnail for: The Cloud, an Arduino, an Application, Server, and me - Adventures In and Out of the Cloud

The Cloud, an Arduino, an Application, Server, and me - Adventures In and Out of the Cloud

Holly Cummins

free
Thumbnail for: TestOps - Chasing the White Whale

TestOps - Chasing the White Whale

Ioana Serban

free
Thumbnail for: Let’s Get to the Rapids: Java 8 Stream Perfomance

Let’s Get to the Rapids: Java 8 Stream Perfomance

Maurice Naftalin

free
Thumbnail for: The Future of Personal Robotics

The Future of Personal Robotics

Nicolas Rigaud

free
Thumbnail for: How to lower the power consumption of your app, without affecting performance

How to lower the power consumption of your app, without affecting performance

Rick Schwartz

free
Thumbnail for: Gimme Caching - The Distributed JCache(JSR107) Way

Gimme Caching - The Distributed JCache(JSR107) Way

David Brimley

free
Thumbnail for: Web Application Security Reloaded for the HTML5 era

Web Application Security Reloaded for the HTML5 era

Carlo Bonamico

free
Thumbnail for: NOW TV and Linear Streaming: The unpredictable scalability challenge

NOW TV and Linear Streaming: The unpredictable scalability challenge

Tom Maule

free
Thumbnail for: Never mind the bollocks: here's the Linux containers

Never mind the bollocks: here's the Linux containers

Patrick Chanezon

free