A Journey into Extensible Effects in Scala

A Journey into Extensible Effects in Scala

This article is an introduction to using the Scala Eff library, which is an implementation of Extensible Effects. This library is now under the Typelevel umbrella, which means it integrates well with popular functional programming libraries in Scala like Cats and Monix. I will not touch on the theoretical side of the concept in this post. Instead, I will be using code snippets to describe how you would introduce it to an existing Scala code base. This should hopefully improve extensibility and maintainability of the code. As part of this, I will demonstrate how to build a purely functional program in Scala using concepts such as Either, Writer and Reader.



I was first introduced to this concept by my colleague Ben Hutchison from his talk at Compose Conference 2016. Since then I have had the opportunity to use Eff in a new project with him in REA.

Purely functional?

In pure functional programming, we model all possible effects of a function in its type signature. This means that functions we call should not have any surprises! No printing to stdout, no logging and no throwing exceptions unless specified in the return type.

Let's demonstrate this using the very common problem of error handling.

Enter the Either monad

The Either monad represents the possibility of failure. Consider these two functions:

def getUser(id: UserId): Either[Error, User] =
  if (id.get > 1000)
    Right(User("Bob", id, PropertyId(123)))
  else
    Left(Error(s"Id ${id.get} in invalid range"))

def getProperty(id: PropertyId): Either[Error, Property] =
  if (id.get > 1000)
    Right(Property("Big house!", id))
  else
    Left(Error("Wrong URL!"))

In a real world scenario, getUser and getProperty may be querying a database instead.

These two functions compose nicely using flatMap or its for comprehension syntactic sugar equivalent as such:

def getPropertyForUserId(id: UserId): Either[Error, Property] =
  for {
    user <- getUser(id)
    property <- getProperty(user.propertyId)
  } yield property

Scala's for comprehension allows us to call getUser and then getProperty if the result of getUser is a Right (success). The final result of all this is a Property if there are no errors or an Error if there is one (e.g. SQLException).

What if we add another monad?

This for comprehension only works if the return type of each expression is of the same monad. What if getProperty needs to make an API call to retrieve a Property, given a URL that we want to provide via an environment variable?

Here is a new version of getProperty with pure dependency injection using the Reader monad (provided by the Cats library). For example, an expression of type Reader[DatabaseConfig, User] can be thought of as an expression that yields a User once a DatabaseConfig is provided. This expression can be used later on by running the Reader, providing a DatabaseConfig.

Let's have a look at how Reader can be introduced into our getProperty code.

def getProperty(id: PropertyId): Reader[PropertyApiUrl, Either[Error, Property]] =
  Reader {
    propertyApiUrl =>
      if (propertyApiUrl.get == "https://production.property-api.com")
        Right(Property("Big house!", id))
      else
        Left(Error("Wrong URL!"))
  }

Now our for comprehension breaks! getUser and getProperty no longer share the same outer monad, so we cannot flatMap over them. Instead, we need to do the following:

def getPropertyForUserId(id: UserId): Either[Error, Property] = {
  val errorOrUser = getUser(id)
  errorOrUser map { user =>
    val readProperty: Reader[PropertyApiUrl, Property] = getProperty(user.propertyId)
    readProperty.run("https://production.property-api.com")
  }
}

The complexity increases the more monads we add. What if we want to do pure logging (which is often achieved using the Writer data type)? What if we introduce asynchrony using Future or monix.eval.Task?

Cleaning it up with Eff

Remember, this problem largely exists because we cannot flatMap over functions whose return types are of a different outer monad.

The Scala Eff library allows us to get around this problem by lifting all these monads into an all-powerful Eff monad. In a nutshell, the Eff monad is able to behave like a combination of other monads that we have seen. These behaviours are described as effects in the vocabulary of Eff.

A program of type Eff[R, A] is a program that contains an R stack of effects (Reader, Either, etc.) and A is its final value after interpreting the entire program.

Let's see how getPropertyForUserId would look if our program was written using the Eff monad.

You will see the usage of the Kind Projector plugin from here on to support partially applied type constructors. For instance, Either[String, ?] is an Either type constructor with the first parameter applied to String and the second parameter unapplied.

type AppStack = Fx.fx2[Either[Error, ?], Reader[PropertyApiUrl, ?]]

def getPropertyForUserId(id: UserId): Either[Error, Property] = {

  val program: Eff[AppStack, Property] = for {
    user <- getUser[AppStack](id)
    property <- getProperty[AppStack](user.propertyId)
  } yield property

  val result: Either[Error, Property] = program
    .runReader(PropertyApiUrl("https://production.property-api.com"))
    .runEither[Error]
    .run

  result match {
    case Left(e) => println(e.msg) // log errors
    case Right(p) => println(s"User ${id.get} owns Property ${p.id.get}")
  }

  result
}

Let's go through this program a snippet at a time.

1. Declare effects in our program

We begin by specifying a union of all the effects our program can have as AppStack. In this case, we have Either[Error, ?] and Reader[PropertyApiUrl, ?]. We define AppStack as such:

type AppStack = Fx.fx2[Either[Error, ?], Reader[PropertyApiUrl, ?]]

2. Construct our program from smaller programs

We call getUser and getProperty just as we did earlier. This gives us one single Eff program that has not executed:

val program: Eff[AppStack, Property] = for {
  user <- getUser[AppStack](id)
  property <- getProperty[AppStack](user.propertyId)
} yield property

This program can be thought of as an Eff expression containing AppStack set of effects. When the expression will provide us a Property when it is eventually interpreted.

3. Interpret our program

We need to then interpret the entire program as follows:

  1. Provide a PropertyApiUrl to describe how to interpret the Reader effect (.runReader(PropertyApiUrl("https://production.property-api.com"))),
  2. Describe how the Either effect should be interpreted. In this case, we print the error to stdout (.runEither[Error] then later on println on the Left case),
  3. Execute the entire program using run. This can only be done after having interpreted all the effects in the stack, otherwise get a compilation error.

It is only at the point of run do we actually execute the program.

Declaration vs Execution (a detour)

As seen above, Eff allows us to compose multiple Eff expressions into one big Eff expression (using flatMap or for comprehension). This Eff expression is merely a description of what the program would do, if you ran it. But why is this useful?

Unit testing

From working with Eff in the past several months, one big advantage I can find in this is being able to interpret your program differently in your unit tests. If the intention to log was captured as an effect in your program, you can use an actual logging framework (e.g. log4j) to interpret the effect in your production code. On the other hand, when interpreting this effect in your unit tests, you may want to println to stdout, or discard the logs so as not to pollute your test output. We will briefly go through incorporating Writer later.

Back to our journey into Eff!

So we have seen how the top level getPropertyForUserId is affected. Now you may ask, what have we done to getUser and getProperty to support this?

getUser with Either effect

type _either[R] = MemberIn[Either[Error, ?], R]

def getUser[R: _either](id: UserId): Eff[R, User] =
  if (id.get > 1000)
    right(User("Bob", id, PropertyId(123)))
  else
    left(Error(s"Id ${id.get} in invalid range"))

We declare a type alias called _either[R], which indicates that Either[Error, ?] is an effect that is a member in an arbitrary stack R of effects.

The implementation of getUser does not change much. The first obvious change is in the return type, which is now Eff[R, User]. We have also constrained R to be a stack of effects that contain an Either[Error, ?] effect. This is described by the R: _either constraint or context bound.

Instead of returning User in a Right, we lift User into the Eff monad using the built-in EitherCreation.right function. For the error scenario, we lift the Error using EitherCreation.left.

Let's look at getProperty now, which requires similar changes.

getProperty with Either and Reader effects

type _readerUrl[R] = MemberIn[Reader[PropertyApiUrl, ?], R]

def getProperty[R: _either : _readerUrl](id: PropertyId): Eff[R, Property] =
  for {
    propertyApiUrl <- ask[R, PropertyApiUrl]
    property <- if (propertyApiUrl.get == "https://production.property-api.com")
      right(Property("Big house!", id))
        else
      left(Error("Wrong URL!"))
  } yield property

First and foremost, we declare a new type alias _readerUrl[R], which is used by getProperty. This type alias indicates that Reader[PropertyApiUrl, ?] is a member in an arbitrary stack R of effects. Compare this with the previous declaration of _either[R].

Similarly, getProperty returns an Eff[R, Property], where R is again constrained to contain an Either[Error, ?] and a Reader[PropertyApiUrl, ?], denoted by R: _either : _readerUrl.

Given that this function contains a Reader[PropertyApiUrl, ?] effect, it is then able to ask for a PropertyApiUrl from the stack R, which it then passes into the PropertyApiService before lifting the results into an Eff[R, Property] using EitherCreation.left and EitherCreation.right again.

The caller of these methods (getPropertyForUserId) must set the type parameter R to be a stack of effects that contains Either[Error, ?] and Reader[PropertyApiUrl, ?]. In our example here, this is provided in the type alias type AppStack = Fx.fx2[Either[Error, ?], Reader[PropertyApiUrl, ?]]. This is used when composing our two functions getUser and getProperty, shown below again:

val program: Eff[AppStack, Property] = for {
  user <- getUser[AppStack](id)
  property <- getProperty[AppStack](user.propertyId)
} yield property

Extension #1: Adding Writer and another Reader

One strong benefit of Eff is its ability to extend. You can add new effects easily without affecting parts of the program that do not care about the effects.

To demonstrate this, let's add a new step in our program to log the current time. Here we are using Reader to read in the time and Writer to signify the intent to log. The actual logging will be done during interpretation of the program.

type _logger[R] = MemberIn[Writer[String, ?], R]
type _readClock[R] = MemberIn[Reader[Instant, ?], R]

def logTime[R : _logger : _readClock](): Eff[R, Unit] =
  for {
    time <- ask[R, Instant]
    _ <- tell(s"The current time is $time")
  } yield ()

What logTime does here is it first asks for an Instant. Subsequently, it uses tell to indicate the intention to log a message.

This changes our top level program, but luckily not by much. We need to change the top level AppStack to include our new effects. Then we need to interpret our Eff program to pass in a time and also to do something about our intention to log!

type AppStack = Fx.fx4[
  Either[Error, ?], Reader[PropertyApiUrl, ?],
  Writer[String, ?], Reader[Instant, ?]
] // Extended to contain Writer and Reader

def getPropertyForUserId(id: UserId): Either[Error, Property] = {

  val program: Eff[AppStack, Property] = for {
    user <- getUser[AppStack](id)
    property <- getProperty[AppStack](user.propertyId)
    _ <- logTime[AppStack]() // Call our new function
  } yield property

  val result: Either[Error, Property] = program
    .runReader(PropertyApiUrl("https://production.property-api.com"))
    .runEither
    .runReader(() => Instant.now())
    .runWriterUnsafe[String] {
      case log => println(log) // print log message to stdout
    }.run

  result match {
    case Left(e) => println(e.msg) // log errors
    case Right(p) => println(s"User ${id.get} owns Property ${p.id.get}")
  }

  result
}

getUser and getProperty can stay exactly as they are! Now when getPropertyForUserId is called with User(1200), it will print the following to stdout:

The current time is 2017-05-31T04:03:49.163Z
User 1200 owns Property 123

Extension #2: Asynchrony

Eff supports Scala Future, Monix Task and Scalaz Task (among others) to handle asynchony in your program. I have put this effect into its own section because asynchonous effects are handled slightly differently in Eff. I assume most of you are more familiar with Scala Future than the others. Eff mandates the interpretation of the Future effect to the very last step.

Let's demonstrate this by changing getUser to delegate to a UserRepository that returns a Future. This is what have to do when using a Scala library that works with Future.

def getUser[R : _either : _Future](id: UserId): Eff[R, User] =
  for {
    errorOrUser <- fromFuture(UserRepository.get(id))
    user <- fromEither(errorOrUser)
  } yield user

Note that the _Future type alias comes with the Eff library. By now you might realise that we need to change the top level of our program to interpret any new effects we introduce to our program. This is how the top level looks like now:

type AppStack = Fx.fx5[
  Either[Error, ?], Reader[PropertyApiUrl, ?],
  Writer[String, ?], Reader[Instant, ?],
  TimedFuture
] // Extended to contain TimedFuture

def getPropertyForUserId(id: UserId): Either[Error, Property] = {

  // this remains unchanged
  val program: Eff[AppStack, Property] = for {
    user <- getUser(id)[AppStack]
    property <- getProperty(user.propertyId)[AppStack]
    _ <- logTime[AppStack]()
  } yield property

  val effFuture: Eff[Fx.fx1[TimedFuture], Either[Error, Property]] = program
    .runReader(PropertyApiUrl("https://production.property-api.com"))
    .runEither
    .runReader(Instant.now())
    .runWriterUnsafe[String] {
      case log => println(log) // print log message to stdout
    }

  val future: Future[Either[Error, Property]] = FutureInterpretation.runAsync(effFuture)

  val result = Await.result(future, Duration("3s"))

  result match {
    case Left(e) => println(e.msg) // log errors
    case Right(p) => println(s"User ${id.get} owns Property ${p.id.get}")
  }

  result
}

There are two differences here from what we previously had. AppStack has been extended to include our new TimedFuture effect, which comes with the Eff library.

When interpreting our program, instead of using run, we call FutureInterpretation.runAsync. This yields a Future that we can wait on as per usual.

Sample code

I have compiled the code in this post including both extensions into a Scastie snippet that you can fork and play around with. Have a look at the Scastie build settings too to see the plugins and libraries used.

My experience

The team I am in started working with Eff earlier this year for our new API. We started building the API without Eff and eventually faced the problem of having to compose stacks of monads. This led to having nested maps and awkward pattern matching as described earlier on in this post. It was at this point that Ben suggested the usage of Eff.

Being fairly new to functional programming, this was my first exposure to building a purely functional web app. It did not take me long to be productive in using Eff, but I was fortunate to be around people who were able to give me a helping hand. Needless to say, I came across a few things in my journey.

Get used to type classes

The first thing I really had to get used to was instead of the effects in my functions being specified in the return type as such: def mean(nums: List[Double]): Option[Double], I had to start reading the effects as constraints on an arbitrary stack R that when evaluated returns a value, like this: def mean[R: _option](nums: List[Double]): Eff[R, Double].

To learn more about type classes and their usefulness, I recommend the Cats tutorial.

Error messages can be cryptic

If you forget to interpret an effect from your program before calling run or runAsync, you get an error like:

Error:(49, 42) No instance found for Member[monix.eval.Task, org.atnos.eff.Fx2[monix.eval.Task,[B]cats.data.Kleisli[[A]A,Config,B]]].
The effect monix.eval.Task is not part of the stack org.atnos.eff.Fx2[monix.eval.Task,[B]cats.data.Kleisli[[A]A,Config,B]]
or it was not possible to determine the stack that would result from removing monix.eval.Task from org.atnos.eff.Fx2[monix.eval.Task,[B]cats.data.Kleisli[[A]A,Config,B]]
Task.fork(TaskInterpretation.runAsync(async), config.scheduler).runAsync(config.scheduler)

It could take you some time before you realise that this means you need to interpret a Reader[Config, ?] effect that exists in your Eff program before you are able to call runAsync to interpret a monix.eval.Task effect.

Eff could be challenging for a team new to functional programming

Eff brings in several concepts that may be new to a team, such as type classes, separating the declaration and execution of a program and writing monadic code. I believe these concepts are incredibly useful and timeless (a monad will always be a monad!), but a team might take a long time to be productive without the presence of someone who is already familiar with these concepts.

Use Eff in your entire program

We started off having parts of our API that are not in Eff, which means we had to lift these functions into Eff (using fromEither, fromOption, etc.). Eventually we refactored these functions to Eff and they became much easier to read. I would suggest having most of your application written as Eff expressions to make everything compose better.

The community is lovely

Being new to the library, I asked a lot of questions in the Gitter channel. The maintainers and other users helped me out a lot. I raised a bug at the end of a work day and it was fixed before I even got home!

Should your team adopt Eff?

Eff comes with many benefits, but it does require spending some time to learn a new technique. However, this isn't just a fancy new library that someone thought would be fun to implement and use. It is backed by research (Extensible Effects and Freer Monads, More Extensible Effects)! For better understanding of the topic, I also recommend viewing the library author Eric Torreborre's talk on this topic in 2016. If you are interested in getting hands-on with Eff, check out Ben Hutchison's series of exercises on Getting Work Done With Extensible Effects.

I do think it is important to acknowledge the time needed to spend on learning a new library and introducing it to the rest of the team before incorporating it in a team-based environment though. We went through this journey in our team and I urge everyone to give this a try if given the opportunity!

Appendix

Monad Transformers

Some readers might see similarities between Eff and Monad Transformers. What I have covered so far is using Eff as a replacement for Monad Transformers. Unlike Monad Transformers, you can combine two Eff programs with different sets of effects to form one giant Eff program. In Monad Transformers, this is much harder. You would need to choose one stack and lift both programs into it.

Free monads

Other parts of REA have been using Free monads and the app interpreter pattern in building their applications for the past couple of years. The idea is to create a DSL that describes your program and writing an interpreter to run the program later on. Similarly separating construction of the program and its execution. Chris Myers discusses this in his talk A Year Living Freely at Typelevel Summit in 2016.

Eff allows you to do this too by writing custom effects. You can use this approach to create a DSL for your program in the same vein as Free monads and writing interpreters for these custom effects. The user guide contains a tutorial to do so.