Jan 9, 2023

Rex!

A simple and intuitive data validation library for Elixir

In all my musings on an independent framework I got hung up on parameter validation.

One of the big things that spawned the idea in me to build my own framework was that I wanted to turn this:

def create_article(conn, %{"author" => author, "content" => content}) do
	with {:ok, article} <- Library.create_article(author, content)
end

Into this:

def create_article(conn, %CreateArticle{author: author, content: content}) do
	with {:ok, article} <- Library.create_article(author, content)
end

In essence, I wanted a way to validate my input parameters, and automatically throw an informative error when they were invalid

Now, in Phoenix the relatively common way to handle this is with Ecto changesets:

# Here I define a custom embedded schema to model
# my input parameters
# I use a :validate action when casting to a schema,
# but this is pure personal prefence
defmodule CreateRequest do
  use Ecto.Schema
  import Ecto.Changeset
  embedded_schema do
    field :first_name, :string
    field :last_name, :string
    field :birth_date, :integer
  end
  @spec new(map) :: {:ok, %__MODULE__{}} | {:error, Changeset.t}
  def new(attrs) do
    %__MODULE__{}
    |> cast(attrs, [:first_name, :last_name, :birth_date])
    |> validate_required([:first_name, :last_name, :birth_date])
    |> apply_action(:validate)
  end
end
# Here is a controller function that then uses the
# schema we created
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, %CreateRequest{first_name: first_name, last_name: last_name, birth_date: birth_date}} <- CreateRequest.new(raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

This is how I had been doing it for a while. But I found it to be extremely verbose, to the point of making my controller files more difficult to read and write. Just looking at a schema file, it’s hard at a glance to discern what actual data you want your controller function to accept. The base type of each field defined right at the top, which is nice, but there are a bunch of extra rules defined lower down across multiple function calls. Moreover, because the data’s type and validations are defined in multiple places, it’s easy to forget to update your validations in all the right places.

Coming from the Python world, my mind immediately went to the wonderful experience I had with Pydantic and FastAPI. I’m personally a huge fan of Pydantic, I think it’s one of the most intuitive software libraries I’ve ever used. It allows you to define Python data structures with types, similar to dataclasses, except whereas dataclasses are only for type annotations, Pydantic models are validated at runtime. This means you can do arbitrarily complex validation while still retaining a clean, declarative API. Pydantic is really cool on it’s own, but the cherry on top is how you can combine it with FastAPI to achieve complex parameter validation with essentially no manual work. And as a cool bonus, input that fails validation will be automatically handled by the framework to return well-structured 422 Unprocessable Entity error responses.

So armed with this goal in mind, I took to figuring out a more ergonomic way to validate my controller params.

My first approach was macros.

• Failed because macros are not composable, don’t easily support ad-hoc validation, and kinda obfuscate validation logic

Then I stumbled upon this excellent post from Sasa Juric:

Towards Maintainable Elixir: The Core and the InterfaceIn my previous article, I presented the development process used at Very Big Things.https://medium.com/very-big-things/towards-maintainable-elixir-the-core-and-the-interface-c267f0da43

In it he describes his teams approach to paramater validation using Ecto’s schemaless changeset feature. This removes the ceremony of our schema to something more like this:

# Here I define a more "functional" schema
def create_schema(params) do
	types = %{
		first_name: :string,
		last_name: :string,
		birth_date: :string
	}
	{%{}, types}
	|> Changeset.cast(params, Map.keys(types))
	|> Changeset.validate_required([:first_name, :last_name, :birth_date])
	|> Changeset.apply_action(:validate)
end
# Because there is no struct involved, we just
# get a map back
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, %{first_name: first_name, last_name: last_name, birth_date: birth_date}} <- create_schema(raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

This certainly helps with making controllers easier to read, and it makes the schema definitions themselves a bit easier to understand as well. However, we still have the issue of defining our data types and validations in multiple places, which isn’t ideal.

Sasa recognized this as well, and so his team instead uses a small wrapper to make schema definitions a bit less noisy.

# HELPERS
# Here we define a common "parse" function that can be used in any controller
# with any provided schema
def parse(schema, params) do
	{%{}, ...extract types...}
	|> Changeset.cast(params, ...extract keys...)
	|> Changeset.validate_required(...required keys...)
	|> Changeset.apply_action(:validate)
end
# CONTROLLER
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
	schema = [
		first_name: [:string, required: true],
		last_name: [:string, required: true],
		birth_date: [:string, required: true]
	]
  with(
		{:ok, [first_name: first_name, last_name: last_name, birth_date: birth_date]} <- parse(schema, raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

Now our schema definition is nice and concise. We only need to look in 1 place to understand exactly what parameters are controllers need, and it’s conveniently co-located directly with the handler itself

What’s interesting about this last approach is that it’s declarative. Rather than defining our validations with functions, we are now defining them with data.

And it’s here I will insert the old Clojure adage:

data > functions > macros

• Jose also mentions this in the official Elixir guide as well, specifically in reference to data validation no less

Domain-specific languagesMeta-programming in Elixir Domain-specific languages (DSL) allow developers to tailor their application to a particular domain.https://elixir-lang.org/getting-started/meta/domain-specific-languages.html

The advantage of defining our validations with data is that we have the full power of Elixir at our fingertips to build validation data concisely and dynamically

So for example, we can naturally compose validations:

# This would usually be a bit more obfuscation to compose a
# schema than I'd like, but it illustrates the point
@required [required: true]
@valid_name [@required, min_length: 2, max_length: 120]
@create_schema [
	first_name: [:string | @valid_name],
	last_name: [:string | @valid_name],
	birth_date: [:string | @required]
]
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, [first_name: first_name, last_name: last_name, birth_date: birth_date]} <- parse(@create_schema, raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile)
	end
end

Or we can compose schemas themselves:

# Again probably don't wanna compose like this in practice
# but it illustrates the point
@update_name_schema [
	first_name: [:string, required: true]
	last_name: [:string, required: true]
]
@create_schema @update_name_schema ++ [
	birth_date: [:string | required: true]
]
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, [first_name: first_name, last_name: last_name, birth_date: birth_date]} <- parse(@create_schema, raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile)
	end
end

We can also fallback on more complex, function-based validations, but now they are as-needed, rather than the default:

# This would usually be a bit more obfuscation to compose a
# schema than I'd like, but it illustrates the point
@valid_name [required: true, min_length: 2, max_length: 120]
@create_schema %{
	first_name: [:string, required: true],
	last_name: [:string, required: true],
	phone_number: [:string, required: true, satisfies: &e164_format/1]
}
@spec e164_format(String.t) :: boolean
defp e164_format(str) do
	...validate str is a valid E164 phone number...
end
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, [first_name: first_name, last_name: last_name, birth_date: birth_date]} <- parse(@create_schema, raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

Enter Rex

This brings me to Rex. Rex is basically just an expansion of this idea, and brings with it a standard grammar that is unobtrusive and easily extensible for your own needs.

With Rex, we can define our schema like this:

@create_schema [
	first_name: :string,
	last_name: :string,
	birth_date: :string
]
@spec create(Plug.Conn.t, map) :: Plug.Conn.t
def create(conn, raw_params) do
  with(
		{:ok, [first_name: first_name, last_name: last_name, birth_date: birth_date]} <- Rex.cast(@create_schema, raw_params),
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

• Parameters are required by default, and if we only wanna cast to a type we can simply provide an atom saying what data type we expect

Here’s a quick cheat sheet of different schema’s you can create with Rex

@create_person [
	phone_number: :string,
	
	# Optional params are denoted with a "?"
	# When casted, the ? will automatically be removed
	first_name?: :string
	# Validation rules support many Elixir idioms
	# Check the docs for a full breakdown
	last_name: [:string, length: 3..120, charset: ~r/^[a-zA-Z]*$/]
	# All cast-able Ecto types are supported
	birth_date: :integer
	account_id: :uuid
	aliases: {:list, :string}
	
	# You can also create your own custom data types,
	# and define custom rules for them
	phone_number: [ExPhoneNumber.Model.PhoneNumber, country_code: 1, is_possible: true]
]

Additionally, because Rex was originally built for parsing Phoenix controller params, we provide a custom plug to handle the validation/error handling logic for you:

@create_schema [
	first_name: :string,
	last_name: :string,
	birth_date: :string
]
@update_schema [
	first_name?: :string,
	last_name?: :string,
	birth_date?: :string
]
plug Rex.Plug,
	create: @create_schema,
	update: @update_schema

# Notice our second paremeter is now a keyword list
# instead of a map
@spec create(Plug.Conn.t, Keyword.t) :: Plug.Conn.t
def create(conn, first_name: first_name, last_name: last_name, birth_date: birth_date) do
  with(
    {:ok, profile} <- App.create_profile(conn.assigns[:account_id], first_name, last_name, birth_date)
	)
  do
    render(conn, "show.json", profile: profile, phone_number: phone_number)
	end
end

• This will automatically trigger your controllers fallback_controller when an invalid set of params are given. By default, this means all invalid requests will receive a well-structured 422 Unprocessable Entity