Introduction to Functional Programming in F# – Part 8

Introduction

In this post we are going to look at adding validation to the code we worked on in Part 6. We will use active patterns that we looked at in the last post (Part 7) and we will see how you can easily model domain errors.

Setting Up

We are going to use the code from Part 6.

Solving the Problem

The first thing we need to do is create a new record type that will store our validated data. Add this type below the Customer type definition:

type ValidatedCustomer = {
    CustomerId : string
    Email : string option
    IsEligible : bool
    IsRegistered : bool
    DateRegistered : DateTime option 
    Discount : decimal option
}

Have a look at the source data to understand why some of the parts are optional.

We now need to add a new helper function to create a ValidatedCustomer:

let create customerId email isEligible isRegistered dateRegistered discount =
    {
        CustomerId = customerId
        Email = email
        IsEligible = isEligible
        IsRegistered = isRegistered
        DateRegistered = dateRegistered
        Discount = discount
    }

Now we need to think about how we handle validation errors. The obvious choices are using Option but then we lose the reason for the error or Result but that is going to make using our new create function difficult to use; We will use Result. The other thing we need to consider is what types of errors do we expect. The obvious ones are missing data (empty string) or invalid data (string to DateTime/decimal/boolean). We also want to return all of the errors, not just the first one, so we need a list of errors in the output. Lets create the error type as a discriminated union with tupled data:

type ValidationError =
| MissingData of name: string
| InvalidData of name: string * value: string

For missing data, we only need the name of the item but for invalid data, we want the name and the value that failed.

Now we will create some helper functions using active patterns to handle empty string, email regex and booleans:

let (|ParseRegex|_|) regex str =
   let m = Regex(regex).Match(str)
   if m.Success then Some (List.tail [ for x in m.Groups -> x.Value ])
   else None

let (|IsValidEmail|_|) input =
    match input with
    | ParseRegex ".*?@(.*)" [ _ ] -> Some input
    | _ -> None

let (|IsEmptyString|_|) (input:string) =
    if input.Trim() = "" then Some () else None

let (|IsBoolean|_|) (input:string) =
    match input with 
    | "1" -> Some true 
    | "0" -> Some false
    | _ -> None

You will need to add 'open System.Text.RegularExpressions' in the declarations at the top of the file.

Now lets create our validate functions using our new ValidationError discriminated union:

let validateCustomerId customerId = // string -> Result
    if customerId <> "" then Ok customerId
    else Error (MissingData "CustomerId")

let validateEmail email = // string -> Result
    if email <> "" then
        match email with
        | IsValidEmail _ -> Ok (Some email)
        | _ -> Error (InvalidData ("Email", email))
    else
        Ok None

let validateIsEligible (isEligible:string) = // string -> Result
    match isEligible with 
    | IsBoolean b -> Ok b 
    | _ -> Error (InvalidData ("IsEligible", isEligible))

let validateIsRegistered (isRegistered:string) = // string -> Result
    match isRegistered with 
    | IsBoolean b -> Ok b 
    | _ -> Error (InvalidData ("IsRegistered", isRegistered))

let validateDateRegistered (dateRegistered:string) = // string -> Result
    match dateRegistered with
    | IsEmptyString -> Ok None
    | _ ->
        let (success, value) = dateRegistered |> DateTime.TryParse 
        if success then Ok (Some value)
        else Error (InvalidData ("DateRegistered", dateRegistered))

let validateDiscount discount = // string -> Result
    match discount with
    | IsEmptyString -> Ok None
    | _ ->
        try
            discount 
            |> decimal
            |> Some
            |> Ok
        with
        | _ -> Error (InvalidData ("Discount", discount))

We now need to create a validation function. Notice that I have added the expected return type:

let validate (input:Customer) : Result =
    let customerId = input.CustomerId |> validateCustomerId
    let email = input.Email |> validateEmail
    let isEligible = input.IsEligible |> validateIsEligible
    let isRegistered = input.IsRegistered |> validateIsRegistered
    let dateRegistered = input.DateRegistered |> validateDateRegistered
    let discount = input.Discount |> validateDiscount
    create customerId email isEligible isRegistered dateRegistered discount // We have a problem

As noted earlier, we now have a problem - The create function isn't expecting Result types. With the skills and knowledge that we have from this series, we can solve this but not in a very elegant way!

Firstly, we create a couple of helper functions to extract Error and Ok data:

let getError input =
    match input with
    | Ok _ -> []
    | Error ex -> [ ex ]

let getValue input =
    match input with
    | Ok v -> v
    | _ -> failwith "Oops, you should have got here!"

Now we create a list of potential errors, concatenate them and then check to see if there are any:

let validate (input:Customer) : Result =
    let customerId = input.CustomerId |> validateCustomerId
    let email = input.Email |> validateEmail
    let isEligible = input.IsEligible |> validateIsEligible
    let isRegistered = input.IsRegistered |> validateIsRegistered
    let dateRegistered = input.DateRegistered |> validateDateRegistered
    let discount = input.Discount |> validateDiscount
    let errors = 
        [ 
            customerId |> getError;
            email |> getError;
            isEligible |> getError;
            isRegistered |> getError;
            dateRegistered |> getError;
            discount |> getError
        ] 
        |> List.concat
    match errors with
    | [] -> Ok (create (customerId |> getValue) (email |> getValue) (isEligible |> getValue) (isRegistered |> getValue) (dateRegistered |> getValue) (discount|> getValue))
    | _ -> Error errors

Finally, we need to plug the validation into the pipeline:

let parse (data:string seq) = // seq -> seq>
    data
    |> Seq.skip 1
    |> Seq.map parseLine
    |> Seq.choose id
    |> Seq.map validate

If you run the code using 'dotnet run', you should get some typed data as output.

If you want to see how we can solve this in a more idiomatically functional way, have a look at my post on Functional Validation in F# Using Applicatives that I did for the 2019 F# Advent Calendar.

Final Code

This is what you should have ended up with:

open System
open System.IO
open System.Text.RegularExpressions

type Customer = {
    CustomerId : string
    Email : string
    IsEligible : string
    IsRegistered : string
    DateRegistered : string
    Discount : string
}

type ValidatedCustomer = {
    CustomerId : string
    Email : string option
    IsEligible : bool
    IsRegistered : bool
    DateRegistered : DateTime option 
    Discount : decimal option
}

type ValidationError =
| MissingData of name: string
| InvalidData of name: string * value: string

type FileReader = string -> Result

let readFile : FileReader =
    fun path ->
        try
            seq { 
                use reader = new StreamReader(File.OpenRead(path))
                while not reader.EndOfStream do
                    yield reader.ReadLine() 
            }
            |> Ok
        with
        | ex -> Error ex

let parseLine (line:string) : Customer option =
    match line.Split('|') with
    | [| customerId; email; eligible; registered; dateRegistered; discount |] -> 
        Some { 
            CustomerId = customerId
            Email = email
            IsEligible = eligible
            IsRegistered = registered
            DateRegistered = dateRegistered
            Discount = discount
        }
    | _ -> None

let create customerId email isEligible isRegistered dateRegistered discount =
    {
        CustomerId = customerId
        Email = email
        IsEligible = isEligible
        IsRegistered = isRegistered
        DateRegistered = dateRegistered
        Discount = discount
    }

let (|ParseRegex|_|) regex str =
   let m = Regex(regex).Match(str)
   if m.Success then Some (List.tail [ for x in m.Groups -> x.Value ])
   else None

let (|IsValidEmail|_|) input =
    match input with
    | ParseRegex ".*?@(.*)" [ _ ] -> Some input
    | _ -> None

let (|IsEmptyString|_|) (input:string) =
    if input.Trim() = "" then Some () else None

let (|IsBoolean|_|) (input:string) =
    match input with 
    | "1" -> Some true 
    | "0" -> Some false
    | _ -> None

let validateCustomerId customerId =
    if customerId <> "" then Ok customerId
    else Error <| MissingData "CustomerId"

let validateEmail email =
    if email <> "" then
        match email with
        | IsValidEmail _ -> Ok (Some email)
        | _ -> Error (InvalidData ("Email", email))
    else
        Ok None

let validateIsEligible (isEligible:string) =
    match isEligible with 
    | IsBoolean b -> Ok b 
    | _ -> Error (InvalidData ("IsEligible", isEligible))

let validateIsRegistered (isRegistered:string) =
    match isRegistered with 
    | IsBoolean b -> Ok b 
    | _ -> Error (InvalidData ("IsRegistered", isRegistered))

let validateDateRegistered (dateRegistered:string) =
    match dateRegistered with
    | IsEmptyString -> Ok None
    | _ ->
        let (success, value) = dateRegistered |> DateTime.TryParse 
        if success then Ok (Some value)
        else Error (InvalidData ("DateRegistered", dateRegistered))

let validateDiscount discount =
    match discount with
    | IsEmptyString -> Ok None
    | _ ->
        try
            discount 
            |> decimal
            |> Some
            |> Ok
        with
        | _ -> Error (InvalidData ("Discount", discount))

let getError input =
    match input with
    | Ok _ -> []
    | Error ex -> [ ex ]

let getValue input =
    match input with
    | Ok v -> v
    | _ -> failwith "Oops, you should have got here!"

let validate (input:Customer) : Result =
    let customerId = input.CustomerId |> validateCustomerId
    let email = input.Email |> validateEmail
    let isEligible = input.IsEligible |> validateIsEligible
    let isRegistered = input.IsRegistered |> validateIsRegistered
    let dateRegistered = input.DateRegistered |> validateDateRegistered
    let discount = input.Discount |> validateDiscount
    let errors = 
        [ 
            customerId |> getError;
            email |> getError;
            isEligible |> getError;
            isRegistered |> getError;
            dateRegistered |> getError;
            discount |> getError
        ] 
        |> List.concat
    match errors with
    | [] -> Ok (create (customerId |> getValue) (email |> getValue) (isEligible |> getValue) (isRegistered |> getValue) (dateRegistered |> getValue) (discount|> getValue))
    | _ -> Error errors

let parse (data:string seq) =
    data
    |> Seq.skip 1
    |> Seq.map parseLine
    |> Seq.choose id
    |> Seq.map validate

let output data =
    data 
    |> Seq.iter (fun x -> printfn "%A" x)

let import (fileReader:FileReader) path =
    match path |> fileReader with
    | Ok data -> data |> parse |> output
    | Error ex -> printfn "Error: %A" ex

[]
let main argv =
    import readFile @"D:\temp\customers.csv"
    0

Conclusion

In this post we have looked at how we can add validation by using active patterns and how easy it is to add additional functionality into the data processing pipeline.

In the next post we will look at improving the code from the first post we worked on by using more domain terminology.

If you have any comments on this series of posts or suggestions for new ones, send me a tweet (@ijrussell) and let me know.