Presentation On Pickles and Spores: Improving Scala's Support for Distributed Programming

Despite unifying frameworks like Akka, distributed systems often look like a patchwork of special-purpose tools, libraries, and frameworks. For example, the essential task of persisting objects by pickling (or serializing) them to a binary or text representation is typically outsourced to a third-party serialization framework. Many of these frameworks, like Google's Protocol Buffers, lack support by the Scala compiler or integration with the Scala standard library, resulting in suboptimal performance, unsatisfying Scala support, or both. Furthermore, to obtain good performance using one of these frameworks, it's often necessary to roll your own type-specialized custom serializers, requiring significant boilerplate. This talk presents a new pickling framework designed for Scala with a few attractive properties: (1) using the framework requires little to no boilerplate, (2) using Scala's implicit parameters, users can add their own easily-swappable pickle format enabling users to persist

Speakers


PDF: slides.pdf

Slides

On

On pICKLES & sPORES IMPROVING SCALA’S SUPPORT FOR DISTRIBUTED PROGRAMMING HEATHER MILLER Wednesday, June 12, 13

with:

with: PHILIPP HALLER TYPESAFE EUGENE BURMAKO EPFL MARTIN ODERSKY EPFL/TYPESAFE Wednesday, June 12, 13

wHAT IS THIS TALK ABOUT?

wHAT IS THIS TALK ABOUT? Wednesday, June 12, 13

wHAT IS THIS TALK ABOUT?

wHAT IS THIS TALK ABOUT? tributed Making dis gramming pro r in scala easie Wednesday, June 12, 13

This kind of distributed syst

This kind of distributed syst Wednesday, June 12, 13 em

ALso!

ALso! This kind of distributed syst insert social network of your choice here Wednesday, June 12, 13 em

Machines Communicating

Machines Communicating Bottomline: Wednesday, June 12, 13

Machines Communicating

Machines Communicating Bottomline: How can we simplify di stribution at the language-level? Wednesday, June 12, 13

Agenda

Agenda sCALA pICKLING macros & typeclasses Spores Wednesday, June 12, 13

Agenda

Agenda sCALA pICKLING Spores Wednesday, June 12, 13

scala.pickling

scala.pickling Pickles! https://github.com/scala/pickling Wednesday, June 12, 13

What is it?

What is it? PICKLING == SERIALIZATION == MARSHALLING https://github.com/scala/pickling Wednesday, June 12, 13

What is it?

What is it? PICKLING == SERIALIZATION == MARSHALLING very differe nt from java seriali zation https://github.com/scala/pickling Wednesday, June 12, 13

wait, why do we care?

wait, why do we care? https://github.com/scala/pickling Wednesday, June 12, 13

wait, why do we care?

wait, why do we care? Slow! https://github.com/scala/pickling Wednesday, June 12, 13

wait, why do we care?

wait, why do we care? Slow! Closed! https://github.com/scala/pickling Wednesday, June 12, 13

wait, why do we care?

wait, why do we care? Slow! Closed! not serializable exceptions at runtime https://github.com/scala/pickling Wednesday, June 12, 13

wait, why do we care?

wait, why do we care? Slow! Closed! ely make t retroactiv can’ not serializable exceptions at runtime serializable classes https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling fast: Serialization code generated at compiletime and inlined at the use-site. https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling fast: Serialization code generated at compiletime and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling fast: Serialization code generated at compiletime and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling fast: Serialization code generated at compiletime and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! https://github.com/scala/pickling Wednesday, June 12, 13

Enter: Scala Pickling

Enter: Scala Pickling fast: Serialization code generated at compiletime and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! typesafe: Picklers are type-specialized. Catch errors at compile-time! https://github.com/scala/pickling Wednesday, June 12, 13

What does it look like?

What does it look like? https://github.com/scala/pickling Wednesday, June 12, 13

What does it look like?

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling Wednesday, June 12, 13

What does it look like?

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  json._ import  json._ https://github.com/scala/pickling Wednesday, June 12, 13

What does it look like?

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36) https://github.com/scala/pickling Wednesday, June 12, 13

What does it look like?

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36) scala>  res0.pickle res1:  scala.pickling.json.JSONPickle  =   JSONPickle({    "tpe":  "Person",    "name":  "John  Oliver",    "age":  36 }) https://github.com/scala/pickling Wednesday, June 12, 13

it’s pretty fast

it’s pretty fast and... https://github.com/scala/pickling Wednesday, June 12, 13

collections: Time

collections: Time Wednesday, June 12, 13 Benchmarks

Benchmarks

Benchmarks collections: free Mem ory (more is better) Wednesday, June 12, 13

collections: size

collections: size Wednesday, June 12, 13 Benchmarks

geotrellis: time

geotrellis: time Wednesday, June 12, 13 Benchmarks

evactor: time

evactor: time Benchmarks Java runs out of memory Wednesday, June 12, 13

Benchmarks

Benchmarks evactor: time (no java, m ore events) Wednesday, June 12, 13

btw,

btw, that’s just the default behavior... https://github.com/scala/pickling Wednesday, June 12, 13

btw,

btw, that’s just the default behavior... you can really customi ze scala pickling too. https://github.com/scala/pickling Wednesday, June 12, 13

Customizing Pickling

Customizing Pickling Previous examples used default behavior Generated picklers Standard pickle format Pickling is very customizable Custom picklers for specific types Custom pickle format Before we can show these things,let's have a look at the building block of the framework... Wednesday, June 12, 13

Pickler Combinators

Pickler Combinators Elegant programming pearl that comes from functional programming. A composable and "constructive" way to think about persisting data. Compose picklers for simple types to build picklers for more complicated types What is a pickler? simplified ver sion of what’s actually used in scalapickling trait  Pickler[T]  {    //  returns  next  write  position    def  pickle(arr:  Array[Byte],  i:  Int,  x:  T):  Int    //  returns  result  plus  next  read  position    def  unpickle(arr:  Array[Byte],  i:  Int):  (T,  Int) } https://github.com/scala/pickling Wednesday, June 12, 13

Pickler Combinators

Pickler Combinators 1 2 We need 2 things: Picklers for base types fully-implemen ted picklers for some basic types like primitives Functions that combine existing picklers to build compound picklers example: combi nator that takes a Pickler[ T] and returns a Pickler[List[T]] https://github.com/scala/pickling Wednesday, June 12, 13

Pickler Combinators

Pickler Combinators Goal: Build a pickler for pairs (Int, String), combine an Int pickler and a String pickler val  myPairPickler  =  tuple2Pickler(intPickler,  stringPickler) What’s the type? Pickler[T], can pickle objects of type T Can we combine them automatically? Can take intPickler and stringPickler as implicit parameters tuple2Pickler can be an implicit def def pickle(implicit pickler: Pickler[(Int, String)]) = { pickler.pickle((32, “yay!”)) } https://github.com/scala/pickling Wednesday, June 12, 13

Implicit Picklers

Implicit Picklers customize what you pic kle! case  class  Person(name:  String,  age:  Int,  salary:  Int) class  CustomPersonPickler(implicit  val  format:  PickleFormat)  extends  SPickler[Person]  {    def  pickle(picklee:  Person,  builder:  PBuilder):  Unit  =  {        builder.beginEntry(picklee)        builder.putField("name",  b  =>            b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())        builder.putField("age",  b  =>            b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())        builder.endEntry()    } } implicit  def  genCustomPersonPickler(implicit  format:  PickleFormat)  =    new  CustomPersonPickler https://github.com/scala/pickling Wednesday, June 12, 13

Pickle Format

Pickle Format        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling Wednesday, June 12, 13

Pickle Format

Pickle Format output any format!        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling Wednesday, June 12, 13

Pickle Format

Pickle Format example talk to a clojure app Output edn, Clojure’s data transfer format. scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  } toy builder implementation: https://gist.github.com/heathermiller/5760171 Wednesday, June 12, 13

What can be pickled?

What can be pickled? types for which an implicit pickler is in scope if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... can’t (yet): instances of inner classes, Types https://github.com/scala/pickling Wednesday, June 12, 13

What can be pickled?

What can be pickled? types for which an implicit pickler is in scope if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which our framework can generate picklers classes IMPORTANT: The implicit case classes picklers are used in the generat generic classes ion! singleton objects primitives & primitive arrays ... can’t (yet): instances of inner classes, Types https://github.com/scala/pickling Wednesday, June 12, 13

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Wednesday, June 12, 13

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Goal: Wednesday, June 12, 13 Scala 2.11 as target

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Goal: Plan: Scala 2.11 as target 1.0 release within the next few months Integration with sbt, Spark, and Akka, ... SIP for Scala 2.11 Experiment: use Scala-pickling to speed up Scala compiler Wednesday, June 12, 13

And now onto

And now onto something completely different. Wednesday, June 12, 13

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Goal: Plan: Scala 2.11 as target 1.0 release within the next few months Integration with sbt, Spark, and Akka, ... SIP for Scala 2.11 Experiment: use Scala-pickling to speed up Scala compiler Wednesday, June 12, 13

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Wednesday, June 12, 13

Status

Status Release 0.8.0 for Scala 2.10.2 https://github.com/scala/pickling ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) No support for inner classes, yet Goal: Plan: Scala 2.11 as target 1.0 release within the next few months Integration with sbt, Spark, and Akka, ... SIP for Scala 2.11 Experiment: use Scala-pickling to speed up Scala compiler Wednesday, June 12, 13

And now onto

And now onto something completely different. Wednesday, June 12, 13

scala

scala spores! http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

What are they?

What are they? Wednesday, June 12, 13

What are they?

What are they? small units of mobile functional behavior Wednesday, June 12, 13

What are they?

What are they? small units of mobile functional behavior proposed for inclusion in scala 2.11 http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Motivation

Motivation First, some http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Motivation

Motivation Potential hazards when using closures incorrectly: • Memory leaks • Race conditions, due to capturing mutable references • Runtime serialization errors, due to unintended capture of references def  receive  =  {    case  Request(data)  =>        future  {            val  result  =  transform(data)            sender  !  Response(result)        } } not a stable va lue! actor spawns a akka o concurrently future t ss incoming reqs proce http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Serialization

Serialization doesn’t extend serializable case  class  Helper(name:  String) class  Main  {    val  helper  =  Helper("the  helper")    val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {        val  result  =  x  +  "  "  +  helper.toString        println("The  result  is:  "  +  result)    } } Serialization of fun throws a NotSerializableException. Why? http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Spores

Spores an alternative way to create closure-like objects, in a way where the environment is controlled val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } Wednesday, June 12, 13

Spores

Spores an alternative way to create closure-like objects, in a way where the environment is controlled val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2. a closure. Wednesday, June 12, 13

Serialization

Serialization doesn’t extend serializable case  class  Helper(name:  String) class  Main  {    val  helper  =  Helper("the  helper")    val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {        val  result  =  x  +  "  "  +  helper.toString        println("The  result  is:  "  +  result)    } } Serialization of fun throws a NotSerializableException. Why? http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Spores

Spores an alternative way to create closure-like objects, in a way where the environment is controlled val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } Wednesday, June 12, 13

Spores

Spores an alternative way to create closure-like objects, in a way where the environment is controlled val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2. a closure. Wednesday, June 12, 13

Spores

Spores an alternative way to create closure-like objects, in a way where the environment is controlled val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2. a closure. The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either 1. parameters of the closure, or 2. declared in the preceding sequence of local value declarations. Wednesday, June 12, 13

Inspiration

Inspiration Spores are not limited to distributed programming do you have state & asynchrony? Make your APIs safer by expecting a Spore instead of a function By requiring spores in your public APIs you can prevent users from introducing hazards, like race conditions. Make your users more happy by preventing them from shooting themselves in the foot! http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

Status

Status SIP-21 Spores: on docs.scala-lang.org/sips now! Get involved in the discussion! Pull request for Scala 2.11 and Akka 2.2.1 in preparation Integration with Scala-pickling planned http://docs.scala-lang.org/sips/pending/spores.html Wednesday, June 12, 13

?

? qUESTIONS Contact heather.miller@epfl.ch @heathercmiller Wednesday, June 12, 13