Feeling Validated – A different way to validate your input

TL;DR

The following post is a Scala example of validating input into your software.  We use Scala, but you can do this in many other languages like Ruby, JavaScript and even Java!  After trying a more traditional approach we then use a type called Validation to encode our business rules.  It’s a different way of thinking.  It uses the Scalaz library.

Let’s Work

So a card is raised for you to validate some input that is coming into your webapp.  No problem, we know how to do validation right?

You glance at the Acceptance Criteria on the card and see the following:

  • User input will be coming into the system in the form of a key/value map, where the keys and values are both strings.
  • These are the fields we are expecting and the validation rules
    • firstName
      • Cannot be null or empty
    • lastName
      • Cannot be null or empty
    • postCode
      • must be a four digit integer
    • phone
      • must be a number
  • A string must be returned to the client stating "Validation success: $firstName $lastName of postcode $postCodeInt has phone number $phoneNumber" if all the validation rules are met.
  • If any of the validation conditions are not met, an error message will be returned with ALL the errors that occurred during validation, such as "Failed to validate with the following errors: first name was not found, last name was not found, phone must be digits"

Well, that’s looks pretty easy.

Though, it looks like we are going to have to collect all the errors along the way and report the whole bunch of failures, instead of just short-circuiting and reporting them one at a time. Dang.

First attempt

Ok, so our function will need to take a Map of key/values and return a String:

def validate(input: Map[String, String]): String

And I guess we are going to need a buffer to collect our errors into:

def validate(input: Map[String, String]): String = {
    val errors = scala.collection.mutable.ListBuffer[String]()
}

So, let’s get validating:

def validate(input: Map[String, String]): String = {
    val errors = scala.collection.mutable.ListBuffer[String]()
    val firstName = input.getOrElse("firstName", "")
    if (firstName.isEmpty)
        errors += "first name can't be empty"
    val lastName = input.getOrElse("lastName", "")
    if (lastName.isEmpty)
        errors += "last name can't be empty"

    val phone = input.getOrElse("phone", "")

    if (!isDigits(phone))
        errors += "Home phone needs to be digits"

    val postCode = input.getOrElse("postCode", "")
    if (!isDigits(postCode))
       errors += "Post code must be a number"

    if (postCode.length != 4)
        errors+="Post code must be 4 digits long"

    if (errors.isEmpty) {
        createSuccessString(firstName, lastName, postCode, phone)
    } else {
        createErrorString(errors.toList)
    }
}

and because we are good programmers we’ve extracted out some helper functions (used above):

def isDigits(input: String): Boolean = input.matches("^[0-9]+$")
def createSuccessString(firstName: String, lastName: String, postCode: String, phoneNumber: String): String =
    s"Validation succeed: $firstName $lastName of postcode $postCode has phone number $phoneNumber"
def createErrorString(errors: List[String]) = s"Failed to validate with the following errors: " + errors.mkString(", ")

Sweet! Home time, right?

REQUIREMENTS CHANGE, REQUIREMENTS CHANGE!

Grr… The business people have just crashed through the window with some emergency requirements changes.

  • Instead of having a phone input, there will be either a homePhone or a mobilePhone.
  • If only a valid homePhone is supplied use that.
  • If only a valid mobilePhone is supplied use that.
  • If both homePhone and mobilePhone are supplied the homePhone will be used.
  • If neither homePhone nor mobilePhone are supplied an error will be flagged for both types.

Sounds like a stupid requirement, but we will give it a go.  The annoying part is that if there are any errors on validating the home phone we can’t show those errors if the mobile phone is valid.  So it looks like we are going to need separate buffer.

The phone validation part of the code is now:

val homePhone = input.getOrElse("homePhone", "")

var hasHomePhone = false

val phoneErrors = ListBuffer[String]()

if (!isDigits(homePhone)) {
    phoneErrors += "Home phone needs to be digits"
} else {
    hasHomePhone = true
}

val mobilePhone = input.getOrElse("mobilePhone", "")

var hasMobilePhone = false

if (!hasHomePhone) {
    if (!isDigits(mobilePhone)) {
        phoneErrors += "mobile phone needs to be digits"
    } else {
        hasMobilePhone = true
    }
}

if (!hasHomePhone && !hasMobilePhone)
    errors ++= phoneErrors

val phone = if (hasHomePhone) homePhone else mobilePhone

Wow, that’s getting a little ugly.  But we are done. Beer o’clock.

REQUIREMENTS CHANGE, REQUIREMENTS CHANGE

Oh please…

So we have more new requirements, looks like the business people are doing their “job” again.

  • Postcodes must be looked up in the postCodeLookupService which returns a suburb name.
  • The postCodeLookupService requires an integer not a string.
  • The postCodeLookupService will return an optional suburb.
  • If the post code is not found in the postCodeLookupService an error will be shown.

Hmmm… so firstly we will have to parse the postcode string and turn it into a integer.  That’s annoying, because an exception will be thrown if it can’t be parsed, which we are then forced to catch.

So our postcode handling now looks like this:

val postCode = input.getOrElse("postCode", "")
var postCodeInt = 0
var suburb = ""

try {
    postCodeInt = Integer.parseInt(postCode)
    val maybeSuburb: Option[String] = postCodeLookup(postCodeInt)

    if (maybeSuburb.isDefined)
        suburb = maybeSuburb.get
    else
        errors += "You don't live in a valid postcode area"
} catch {
    case e: NumberFormatException => errors += "post code must be a number"
}

I’m not feeling excited by this code.  We are relying on exceptions to control flow, the mutable errors buffer is getting added to all over the place, and we are making a call to an external service right in the middle of our validation code.

So the final hellish looking code is this:

def validate(input: Map[String, String]): String = {
    val errors = scala.collection.mutable.ListBuffer[String]()
    val firstName = input.getOrElse("firstName", "")
    if (firstName.isEmpty)
        errors += "first name can't be empty"
    val lastName = input.getOrElse("lastName", "")
    if (lastName.isEmpty)
        errors += "last name can't be empty"

    val homePhone = input.getOrElse("homePhone", "")
    var hasHomePhone = false
    val phoneErrors = ListBuffer[String]()

    if (!isDigits(homePhone)) {
        phoneErrors += "Home phone needs to be digits"
    } else {
        hasHomePhone = true
    }

    val mobilePhone = input.getOrElse("mobilePhone", "")
    var hasMobilePhone = false

    if (!hasHomePhone) {
        if (!isDigits(mobilePhone)) {
            phoneErrors += "mobile phone needs to be digits"
        } else {
            hasMobilePhone = true
        }
    }

    if (!hasHomePhone && !hasMobilePhone)
        errors ++= phoneErrors
    val phone = if (hasHomePhone) homePhone else mobilePhone
    val postCode = input.getOrElse("postCode", "")
    var postCodeInt = 0
    var suburb = ""
    try {
        postCodeInt = Integer.parseInt(postCode)
        val maybeSuburb: Option[String] = postCodeLookup(postCodeInt)
        if (maybeSuburb.isDefined)
            suburb = maybeSuburb.get
        else
            errors += "You don't live in a valid postcode area"
    } catch {
        case e: NumberFormatException => errors += "post code must be a number"
    }
    if (errors.isEmpty) {
        createSuccessString(firstName, lastName, postCode, phone)
    } else {
        createErrorString(errors.toList)
    }
}

Can we do better?

Introducing the Validation type

A completely different way of thinking about this problem is by using the Validation type.  It is a type that contains either a successful value or a failure value, but never both at the same time.  In Scala it looks something like this:

trait Validation[E,A]

final case class Success[E, A](a: A) extends Validation[E, A]

final case class Failure[E, A](e: E) extends Validation[E, A]

What this means is that Validation is an abstract class with two (and only two) concrete subclasses Success and Failure.

This is the perfect way of representing a possible failure.

The Validation type is provided by the Scalaz library.

Making the world a better place

For starters we can return a Validation type from our validateName function

def validateName(label: String, name: String): Validation[String, String] =
  if (!name.isEmpty)
    Success(name)
  else
    Failure(s"$label can't be empty")

Let’s fire up our REPL and give it a go. If we call validateName with an empty first name we should get a failure:

scala> validateName("first name", "")
res0: scalaz.Validation[String,String] = Failure(first name can't be empty)

If we give it a valid name we should get a success:

scala> validateName("first name", "Elvis")
res1: scalaz.Validation[String,String] = Success(Elvis)

And we do!

We can also convert the lookup of the input map to return a Validation:

def validateInput(input: Map[String, String], key: String):Validation[String,String] = 
    input.get(key).toSuccess(s"Could not find $key")

So if the expected key is in the map we will get a Success back, otherwise we will get a Failure with an error message.

Let’s test it:

scala> validateInput(Map(), "firstName")
res2: scalaz.Validation[String,String] = Failure(Could not find firstName)

and

scala> validateInput(Map("firstName" -> "Elvis"), "firstName")
res3: scalaz.Validation[String,String] = Success(Elvis)

Validations are cool because they can be combined with other validations.  One of the methods to do this is flatMap.

flatMap

flatMap takes a function that returns a Validation as well.  If the first Validation is successful it will then call the function, otherwise the whole thing stops.  This is perfect for our first use case. If we can’t find the firstName in the input map there is no point in validating that the first name is not empty.

val validatedFirstName = validateInput(input, "firstName").flatMap(firstName => validateName("first name", firstName))

We have now chained these two validations together.  If either fail, the whole thing will fail.  Otherwise we have a successful first name.

Let’s test it:

scala> validateInput(Map(), "firstName").flatMap(firstName => validateName("first name", firstName))
res4: scalaz.Validation[String,String] = Failure(Could not find firstName)
scala> validateInput(Map("firstName" -> ""), "firstName").flatMap(firstName => validateName("first name", firstName))
res5: scalaz.Validation[String,String] = Failure(first name can't be empty)
scala> validateInput(Map("firstName" -> "Elvis"), "firstName").flatMap(firstName => validateName("first name", firstName))
res6: scalaz.Validation[String,String] = Success(Elvis)

This is also great for the post code use case.  If the post code is a valid integer then we want to check that it is in the postCodeLookup service.

val validatedSuburb = validateInput(input, "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(nfe => "post code must be a number"))
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area"))

But what is this funny line?

postCode.parseInt.leftMap(nfe => "post code must be a number")

Well, we have a helper method called parseInt which also returns a Validation!  We are then converting the NumberFormatException to a nicer error message with the leftMap (which would be better called failMap).

The code on this line:

postCode => postCodeLookup(postCode).toSuccess("not in a valid area")

converts the response from the postCodeLookup service to a Validation.

So we are guaranteed to either get a failure or an actual suburb!

Let’s try it!

scala> validateInput(Map("postCode" -> ""), "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(nfe => "post code must be a number"))
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area"))
res5: scalaz.Validation[String,String] = Failure(post code must be a number)
scala> validateInput(Map("postCode" -> "9999"), "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(nfe => "post code must be a number"))
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area"))
res7: scalaz.Validation[String,String] = Failure(not in a valid area)
scala> validateInput(Map("postCode" -> "3121"), "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(nfe => "post code must be a number"))
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area"))
res8: scalaz.Validation[String,String] = Success(Richmond)

orElse

Now what about this weird phone thing, where we only want either a homePhone or mobilePhone but not both?  Easy!  orElse.

val mobilePhone = validatePhone(input, "mobilePhone", "mobile phone")
val homePhone = validatePhone(input, "homePhone", "home phone")
val vPhone = mobilePhone orElse homePhone

If the mobilePhone validation fails, we will get the homePhone.  And vice-versa.  If neither exists, we will get an error message.

accumulate

The real win from using Validations is that they can accumulate errors into a list by themselves.  To do this, we are going to change the code a little to use something called a NonEmptyList, which is a type-safe way of say that if you get a failed Validation you are guaranteed to have at least one error message.

We are going to create a function called accumulate, which will take a function that will create our success string as one parameter and all the validations we’ve created as the other parameters.  If all our validations are successful then our function will get called.  Otherwise all the error messages from the supplied validations will be collected in a list.

val successStringOrListOfErrors = accumulate(createSuccessString)
  (validatedFirstName, validatedLastName, validatedSuburb, validatedPhone)

Once we call accumulate we are either going to have a success string or a list of errors.  For our last trick, we need to return a string back to the client.

successStringOrListOfErrors.valueOr(errors => createErrorString(errors.list))

valueOr will either give us back the success string or takes a function that will convert our list of errors to a string.

And we are done!

def validate(input: Map[String, String]): String = {
    val validatedFirstName = validateInput(input, "firstName").flatMap(firstName => validateName("first name", firstName))

    val validatedLastName = validateInput(input, "lastName").flatMap(lastName => validateName("last name", lastName))

    val validatedSuburb = validateInput(input, "postCode")
.flatMap(postCode => postCode.parseInt.leftMap(_ => "post code must be a number").toValidationNel)
.flatMap(postCode => postCodeLookup(postCode).toSuccess("not in a valid area").toValidationNel)

    val mobilePhone = validatePhone(input, "mobilePhone", "mobile phone")
    val homePhone = validatePhone(input, "homePhone", "home phone")
    val validatedPhone = mobilePhone orElse homePhone 
    val successStringOrListOfErrors = accumulate(createSuccessString)
(validatedFirstName, validatedLastName, validatedSuburb, validatedPhone)

    successStringOrListOfErrors.valueOr(errors => createErrorString(errors.list))
}

Conclusion

Our final code is much cleaner and concise.  All the validation functions are independent and can be weaved together however you see fit.  There are no ugly mutable buffers running throughout our code.  We’ve looked at a couple of functions used to combine validations such as flatMap and orElse, but there are many other combinator functions.  As we said, this has been a Scala example, but this holds true for other languages as well.

The whole code can be found in this gist here: https://gist.github.com/cwmyers/e07e706436fe4a7c92ef