Denis Defreyne

Avoiding bugs in Ruby code using the state pattern

2023-01-05T23:00:00Z

“Be more careful” is often not useful advice. Software bugs are inevitable, but as I become proficient in software development, my code tends to have fewer initial bugs.

In some part, this is because experience has taught me to spot bugs more quickly. But more importantly, I’ve learned techniques to structure code in ways that make bugs much less likely to arise in the first place.

One such technique is the state pattern. In this article, I’ll take you through an example, at first with a brittle implementation, and then reworked using the state pattern into something more stable.

The example: A messaging platform

The example I’ll use throughout this article is a simplified messaging platform. On this platform, any person can register:

account = Account.register(account_details)

The account details, passed to the .register method, would include at least an email address, but also properties such as first name and last name.

Before the registered account is usable, the person needs to confirm the account, using a code they received via email:

account.confirm("ABC123")

Then, an administrator is expected to approve the account before it can be used:

account.approve

With the account confirmed (by the person registering) and approved (by an administrator), the person is now free to use their account to send messages to other people:

account.message(
  tim,
  "I’m selling these fine leather jackets.",
)

It is important that people cannot send messages unless their account is both confirmed and approved.¹

Let’s take a look at two different ways of implementing this: a naïve implementation, and a safer implementation that uses the state pattern.

A naïve implementation

The simplest implementation (that I can think of) starts with an initializer:

class Account
  def initialize
    @code = "ABC123"
  end

The initializer sets the confirmation code. In a real-world scenario, the confirmation code would be randomly generated. Here, it is hardcoded for the sake of simplicity.

We’ll also need an implementation for the Account.register method, which we used earlier:

  def self.register(account_details)
    new
  end

For this demo implementation, all it does is create a new instance of Account. In a real-world scenario, this might be the place to create a new database record and send out an email with the confirmation code. For simplicity, all of that is left out.

We’ll need #confirm, which checks whether the given code is correct, and advances the state to :confirmed:

  def confirm(code)
    if code != @code
      raise InvalidConfirmationCode
    end

    @state = :confirmed
  end

Next up is #approve, which advances the state to :confirmed_and_approved:

  def approve
    @state = :confirmed_and_approved
  end

Finally, we have #message, which is the method used for sending messages to other accounts. It requires that the account is in the :confirmed_and_approved state:

  def message(who, text)
    unless @state == :confirmed_and_approved
      raise AccountNotApproved
    end

    puts "Sending message to #{who}"
  end
end

In this example implementation of #message, we just log the message.

We’ll also need these two exception classes:

AccountNotApproved = Class.new(StandardError)
InvalidConfirmationCode = Class.new(StandardError)

Here is an example that ties it all together:

tims_account_id = 12358

account = Account.new
account.confirm("ABC123")
account.approve
account.message(tims_account_id, "What’s up, Tim?")

Here, we confirm the account, then an administrator approves the account, and lastly we use the account to send a message to Tim, whose account ID is 12358. The terminal output now shows this:

Sending message to 12358.

So far, so good.

But what if we confirm the account again, after it has already been approved?

tims_account_id = 12358

account = Account.new
account.confirm("ABC123")
account.approve
account.confirm("ABC123")
account.message(tims_account_id, "What’s up, Tim?")

Running this example results in an error during the call to the #message method:

account.rb:9:in `message':
  AccountNotApproved (AccountNotApproved)

AccountNotApproved?! The account definitely was approved. The issue is that our implementation of #confirm set the state to :confirmed:

  def confirm(code)
    if code != @code
      raise InvalidConfirmationCode
    end

    @state = :confirmed
  end

Here, we have a bug: the state shouldn’t be set to :confirmed if the account is already approved. One quick fix for this would be to zero out the @code variable, so that attempting to confirm again would fail:

  def confirm(code)
    if code != @code
      raise InvalidConfirmationCode
    end

    @state = :confirmed
    @code = ""
  end

That is a bit of a hack, though. A slightly better way to solve this is to only allow confirmation when the @state is the initial state:

  def confirm(code)
    return if @state != :initial

    if code != @code
      raise InvalidConfirmationCode
    end

    @state = :confirmed
  end

We’d also need to set the initial @state in the initializer:

  def initialize
    @code = "ABC123"
    @state = :initial
  end

With that bug solved, let us take a look at another issue: it is possible for an account to be approved without having gone through confirmation at all:

tims_account_id = 12358

account = Account.new
account.approve
account.message(tims_account_id, "What’s up, Tim?")

The terminal output now shows this:

Sending message to 12358.

This might be a bug, or it might not be. Perhaps it is intentional that administrators can approve accounts, skipping confirmation. Or perhaps this is an oversight in our implementation.

If we assume it is an oversight, and thus a bug, one way of fixing it is to verify that the state is what we expect it to be:

  def approve
    raise AccountNotConfirmed if @state != :confirmed

    @state = :confirmed_and_approved
  end

We’ll also need to define a new exception:

AccountNotConfirmed = Class.new(StandardError)
AccountNotApproved = Class.new(StandardError)
InvalidConfirmationCode = Class.new(StandardError)

The code is now — as far as I can tell — bug-free, though I’m not comfortable with the end result. This code feels brittle to me: any change in the future has a high chance of inadvertently modifying the expected behavior.

This could would need an extensive test suite. Such a test suite would verify that going through all the different paths, in all different orders, yield correct results. The presence of a test suite would make me feel more comfortable, but there is more that we can do.

Let’s now take a look at a new implementation, which uses @state rather differently.

A safer implementation

In this implementation, the Account class no longer contains the functionality for confirming, approving, and messaging. Rather, all of that is delegated to @state, which is now its own object:

class Account
  attr_accessor :state

  def initialize
    @state = InitialAccountState.new(self)
  end

  def confirm(code)
    @state.confirm(code)
  end

  def approve
    @state.approve
  end

  def message(who, text)
    @state.message(who, text)
  end
end

You could go fancy and use Ruby’s built-in Forwardable module for that if you want. With that module, the implementation of Account could look like this — doing exactly the same:

class Account
  extend Forwardable
  def_delegators :@state, :confirm, :approve, :message

  attr_accessor :state

  def initialize
    @state = InitialAccountState.new(self)
  end
end

Here is what InitialAccountState looks like:

class InitialAccountState
  def initialize(account)
    @code = "ABC123"
    @account = account
  end

  def confirm(code)
    if code != @code
      raise InvalidConfirmationCode
    end

    @account.state = ConfirmedAccountState.new(@account)
  end
end

The InitialAccountState#confirm method is quite similar to the original #confirm: it checks the confirmation codes, and raises an exception if they don’t match.

While the original #confirm changed the state using @state = :confirmed, this new #confirm method changes the account state to a new state object. (We’ll get to the implementation of ConfirmedAccountState in a bit.)

Also worth noting is that the confirmation code lives in the InitialAccountState instance. That is the only place where it is useful. It could also live in Account, but it wouldn’t have a purpose there.

Without having the other state objects implemented, we can already see that one of the buggy behaviors from earlier no longer silently succeeds:

tims_account_id = 12358

account = Account.new
account.approve
account.message(tims_account_id, "What’s up, Tim?")

This piece of code raises a NoMethodError:

account_with_state.rb:13:
  in `approve': undefined method `approve' for
  #<InitialAccountState:0x0000000100907fa8 …>
  (NoMethodError)

  @state.approve(code)
        ^^^^^^^^
      from account_with_state.rb:66:in `<main>'

This NoMethodError is intentional! It signals that there hasn’t been any thought put into the situation where someone would try to approve an account that hasn’t been confirmed yet.

This NoMethodError is a replacement for undefined behavior. I believe that it is preferable to get a NoMethodError than to execute incorrect behavior.

We are still able to implement #approve here, if we wish. Perhaps it is desirable to approve accounts from their initial state, bypassing confirmation. If so, we could implement #approve as follows:

class InitialAccountState
  …

  def approve
    @account.state = ConfirmedAndApprovedAccountState.new(@account)
  end
end

Let’s move on to ConfirmedAccountState:

class ConfirmedAccountState
  def initialize(account)
    @account = account
  end

  def approve
    @account.state = ConfirmedAndApprovedAccountState.new(@account)
  end
end

The #approve method exists here, and moves the state forward to the “account approved” state.

The ConfirmedAccountState state has no #confirm method here. If we were to try to confirm an already approved account, we’d get a NoMethodError:

account_with_state.rb:28:
  in `confirm': undefined method `confirm' for
  #<ConfirmedAccountState:0x0000000100907fa8 …>
  (NoMethodError)

  @state.confirm(code)
        ^^^^^^^^
      from account_with_state.rb:66:in `<main>'

Here, the NoMethodError is less desirable. I would probably implement #confirm anyway, and have it do nothing:

class ConfirmedAccountState
  …

  def confirm
    # Already confirmed; do nothing
  end

This makes the behavior explicit: confirming an account that is already confirmed does nothing. The #confirm method in our naïve implementation also did nothing in this case, but it wasn’t nearly as explicit.

Lastly, we have ConfirmedAndApprovedAccountState:

class ConfirmedAndApprovedAccountState
  def initialize(account)
    @account = account
  end

  def message(who, text)
    puts "Sending message to #{who}"
  end
end

This state is the least interesting: it just has the #message for sending messages.

For this state, it makes sense to implement the #confirm and #approve methods, and have them do nothing:

class ConfirmedAndApprovedAccountState
  …

  def confirm
    # Already confirmed; do nothing
  end

  def approve
    # Already approved; do nothing
  end

With all that, we have an implementation where the state transitions are explicit, and it is also more clear what actions can be taken in which states.

Closing thoughts

The implementation which uses the state pattern makes me the most comfortable. It avoids undefined behavior, and even though NoMethodErrors are a little nasty, they’re better to have than undefined behavior.

There is nothing preventing us from implementing all methods, and avoiding NoMethodError altogether. For each combination of state and method, we’d have to think about what the behavior should be. Perhaps it is doing nothing, perhaps it is raising a specific exception, or perhaps it is doing something else entirely.

Defining the behavior for each combination of state and method is not always easy. This can make the state pattern look cumbersome and difficult. However, if we want high-quality software, we can’t get away from defining behavior anyway. It seemed to be optional in the original, naïve implementation, but that led to bugs as a result.

I prefer explicit, readable code over compact code every time. Explicit code makes it easier to find and prevent bugs. Explicit, readable code is less likely to break over time, even in a codebase with a large amount of churn.

Back in university, a professor said to me that if statements are a code smell. While I think that is an over-generalisation, you’ll find that the original, naïve implementation has quite some ifs — especially in the bug fixes — while the state pattern implementation has few.

If in the future I find myself in a situation where behavior depends on state, you can be sure I’ll whip out the state pattern.

That should cut down on the cryptocurrency/NFT spam. That is unfortunately still a thing, right? ↩

The complete guide to implementing equality in Ruby

2022-05-25T22:00:00Z

Ruby is one of the few programming languages that gets equality right. I often play around with other languages, but keep coming back to Ruby. In large part, this is because Ruby’s implementation of equality is so nice.¹

Nonetheless, equality in Ruby is not straightforward. There is #==, #eql?, #equal?, #===, and more. Even if you’re familiar with using them, implementing them can be a whole other story.

Let’s walk through all forms of equality in Ruby and how to implement them.

Why implementing equality matters
1. Entities
2. Value objects
Basic equality (double equals)
1. Basic equality for value objects
2. Basic equality for entities
Basic equality with type coercion
Strict equality
Hash equality
Case equality (triple equals)
Ordered comparison
Wrapping up
Further reading

Why implementing equality matters

We check whether objects are equal all the time. Sometimes we do this explicitly, sometimes implicitly. Here are some examples:

Do these two Employees work in the same Team? Or, in code: denis.team == someone.team.
Is the given DiscountCode valid for this particular Product? Or, in code: product.discount_codes.include?(given_discount_code).
Who are the (distinct) managers for this given group of employees? Or, in code: employees.map(&:manager).uniq.

A good implementation of equality is predictable; it aligns with our understanding of equality.

An incorrect implementation of equality, on the other hand, conflicts with what we commonly assume to be true. Here is an example of what happens with such an incorrect implementation:

# Find the book “Gödel, Escher, Bach”
repo = BookRepository.new
geb      = repo.find(isbn: "978-0-14-028920-6")
geb_also = repo.find(isbn: "978-0-14-028920-6")

geb == geb_also # => false?!

The geb and geb_also objects should definitely be equal. The fact that the code says they’re not, is bound to cause bugs down the line. Luckily, we can implement equality ourselves and avoid this class of bugs.

No one-size-fits-all solution exists for an equality implementation. However, there are two kinds of objects where we do have a general pattern for implementing equality: entities and value objects. These two terms come from domain-driven design (DDD for short), but they’re relevant even if you’re not using DDD. Let’s take a closer look.

Entities

Entities are objects that have an explicit identity attribute. Often, entities are stored in some database and have a unique id attribute, corresponding to a unique id table column. The following Employee example class is such an entity:

class Employee
  attr_reader :id
  attr_reader :name

  def initialize(id, name)
    @id = id
    @name = name
  end
end

Two entities are equal when their IDs are equal. All other attributes are ignored. After all, an employee’s name might change, but that does not change their identity. Imagine getting married, changing your name and not getting paid anymore because HR has no clue who you are anymore!

ActiveRecord, the ORM that is part of Ruby on Rails, calls entities models instead, but they’re the same concept. These model objects automatically have an ID. In fact, ActiveRecord models already implement equality correctly out of the box!

Value objects

Value objects are objects without an explicit identity. Instead, their value as a whole constitutes identity. Consider this Point class:

class Point
  attr_reader :x
  attr_reader :y

  def initialize(x, y)
    @x = x
    @y = y
  end
end

Two Points will be equal if their x and y values are equal. The x and y values constitute the identity of the point.

In Ruby, the basic value object types are numbers (both integers and floating-point numbers), characters, booleans, and nil. For these basic types, equality works out of the box:

17 == 17      # => true
false != 12   # => true
1.2 == 3.4    # => false

Arrays of value objects are in themselves also value objects. Equality for arrays of value objects works out of the box — for example, [17, true] == [17, true]. This might seem obvious, but this is not true in all programming languages.

Other examples of value objects are timestamps, date ranges, time intervals, colors, 3D coordinates, and money objects. These are built from other value objects: for example, a money object consists of a fixed-decimal number and a currency code string.

Basic equality (double equals)

Ruby has the == and != operators for checking whether two objects are equal or not:

"orange" == "red"   # => false
1 + 3 == 4          # => true
12 - 1 != 10        # => true

Ruby’s built-in types all have a sensible implementation of ==. Some frameworks and libraries provide custom types, which will have a sensible implementation of ==, too. Here is an example with ActiveRecord:

geb = Book.find_by(title: "Gödel, Escher, Bach")
also_geb = Book.find_by(isbn: "978-0-14-028920-6")

geb == also_geb   # => true

For custom classes, the == operator returns true if and only if the two objects are the exact same instance. Ruby does this by checking whether the internal object IDs are equal. These internal object IDs are accessible using #__id__. Effectively, gizmo == thing is the same as gizmo.__id__ == thing.__id__.

This behavior is often not a good default, however. To illustrate this, consider the Point class from earlier:

class Point
  attr_reader :x
  attr_reader :y

  def initialize(x, y)
    @x = x
    @y = y
  end
end

The == operator will return true only when calling it on itself:

a = Point.new(4, 6)
b = Point.new(3, 7)
a_also = Point.new(4, 6)

a == a        # => true
a == b        # => false
a == "soup"   # => false
a == a_also   # => false?!

This default behavior is often undesirable in custom classes. After all, two points are equal if (and only if) their x and y values are equal. This behavior is undesirable for value objects (such as Point), and entities (such as the Employee class mentioned earlier).

The desired behavior for value objects and entities is as follows:

For value objects, we’d like to check whether all attributes are equal.

For entities, we’d like to check whether the explicit ID attributes are equal.

By default, Ruby checks whether the internal object IDs are equal.

Instances of Point are value objects. With the above in mind, a good implementation of == for Point would look as follows:

class Point
  …
  def ==(other)
    self.class == other.class &&
      @x == other.x &&
      @y == other.y
  end
end

This implementation checks all attributes and the class of both objects. By checking the class, checking equality of a Point instance and something of a different class return false rather than raise an exception.

Checking equality on Point objects now works as intended:

a = Point.new(4, 6)
b = Point.new(3, 7)
a_also = Point.new(4, 6)

a == a        # => true
a == b        # => false
a == "soup"   # => false
a == a_also   # => true

The != operator works too:

a != b          # => true
a != "rabbit"   # => true

A correct implementation of equality has three properties: reflexivity, symmetry, and transitivity.

Reflexivity: An object is equal to itself: a == a.

Symmetry: If a == b, then b == a.

Transitivity: If a == b and b == c, then a == c.

These properties embody a common understanding of what equality means. Ruby won’t check these properties for you, so you’ll have to be vigilant to ensure you don’t break these properties when implementing equality yourself.

IEEE 754 and violations of reflexivity

It seems natural that something would be equal to itself, but there is an exception. IEEE 754 defines NaN (Not a Number) as a value resulting from an undefined floating-point operation, such as dividing 0 by 0. NaN by definition is not equal to itself. You can see this for yourself:

Float::NAN == Float::NAN   # => false

This means that == in Ruby is not universally reflexive. Luckily, exceptions to reflexivity are exceedingly rare; this is the only exception that I am aware of.

Basic equality for value objects

The Point class is an example of a value object. The identity of a value object, and thereby equality, is based on all its attributes. That is exactly what the earlier example does:

class Point
  …
  def ==(other)
    self.class == other.class &&
      @x == other.x &&
      @y == other.y
  end
end

Basic equality for entities

Entities are objects with an explicit identity attribute, commonly @id. Unlike value objects, an entity is equal to another entity if and only if their explicit identities are equal.

Entities are uniquely identifiable objects. Typically, any database record with an id column corresponds to an entity.² Consider the following Employee entity class:

class Employee
  attr_reader :id
  attr_reader :name

  def initialize(id, name)
    @id = id
    @name = name
  end
end

For entities, the == operator is more involved to implement than for value objects:

class Employee
  …
  def ==(other)
    super || (
      self.class == other.class &&
      !@id.nil? &&
      @id == other.id
    )
  end
end

This code does the following:

The super call invokes the default implementation of equality: Object#==. On Object, the #== method returns true if and only if the two objects are the same instance. This super call, therefore, ensures that the reflexivity property always holds.
As with Point, the implementation Employee#== checks class. This way, an Employee instance can be checked for equality against objects of other classes, and this will always return false.
If @id is nil, the entity is considered not equal to any other entity. This is useful for newly-created entities which have not been persisted yet.
Lastly, this implementation checks whether the ID is the same as the ID of the other entity. If so, the two entities are equal.

Checking equality on entities now works as intended:

denis       = Employee.new(570, "Denis Defreyne")
also_denis  = Employee.new(570, "Denis")
denis_v     = Employee.new(992, "Denis Villeneuve")
new_denis_1 = Employee.new(nil, "Denis 1")
new_denis_2 = Employee.new(nil, "Denis 2")

denis == denis               # => true
denis == also_denis          # => true
denis == "cucumber"          # => false
denis == denis_v             # => false
denis == new_denis_1         # => false
new_denis_1 == new_denis_1   # => true
new_denis_1 == new_denis_2   # => false

Blog post of Theseus

Implementing equality on entity objects isn’t always straightforward. An object might have an id attribute that doesn’t quite align with the object’s conceptual identity.

Take a BlogPost class, for example, with id, title, and body attributes. Imagine creating a BlogPost, then halfway through writing the body for it, scratching everything and start over with a new title and a new body. The id of that BlogPost will still be the same, but is it still the same blog post?

If I follow a Twitter account which later gets hacked and turned into a cryptocurrency spambot, is it still the same Twitter account?

These questions don’t have a proper answer. That’s not surprising, as this is essentially the Ship of Theseus thought experiment. Luckily, in the world of computers, the generally accepted answer seems to be yes: if two entities have the same id, then the entities are equal as well.

Basic equality with type coercion

Typically, an object is not equal to an object of a different class. However, this is not always the case. Consider integers and floating-point numbers:

float_two = 2.0
integer_two = 2

Here, float_two is an instance of Float, and integer_two is an instance of Integer. They are equal: float_two == integer_two is true, despite different classes. Instances of Integer and Float are interchangeable when it comes to equality.

As a second example, consider this Path class:

class Path
  attr_reader :components

  def initialize(*components)
    @components = components
  end

  def string
    "/" + @components.join("/")
  end
end

This Path class provides an API for creating paths:

path = Path.new("usr", "bin", "ruby")
path.string   # => "/usr/bin/ruby"

The Path class is a value object, and implementing #== could be done just as with other value objects:

class Path
  …
  def ==(other)
    self.class == other.class &&
      @components == other.components
  end
end

However, the Path class is special because it represents a value that could be considered a string. The == operator will return false when checking equality with anything that isn’t a Path:

path = Path.new("usr", "bin", "ruby")
path.string == "/usr/bin/ruby"   # => true
path == "/usr/bin/ruby"          # => false :(

It can be beneficial for path == "/usr/bin/ruby" to be true rather than false. To make this happen, the == operator needs to be implemented differently:

class Path
  …
  def ==(other)
    other.respond_to?(:to_str) &&
      to_str == other.to_str
  end

  def to_str
    string
  end
end

This implementation of == coerces both objects to Strings, and then checks whether they are equal. Checking equality of a Path now works:

path = Path.new("usr", "bin", "ruby")

path == path                # => true
path == "/usr/bin/ruby"     # => true
path == "/usr/bin/python"   # => false

path == 12.3   # => false

This class implements #to_str, rather than #to_s. These methods both return strings, but by convention, the to_str method is only implemented on types that are interchangeable with strings.

The Path class is such a type. By implementing Path#to_str, the implementation states that this class behaves like a String. For example, it’s now possible to pass a Path (rather than a String) to IO.open, and it will just work. This is because IO.open accepts anything that responds to #to_str.

String#== also uses the to_str method. Because of this, the == operator is reflexive:

"/usr/bin/ruby" == path     # => true
"/usr/bin/python" == path   # => false

Strict equality

Ruby provides #equal? to check whether two objects are the same instance:

"snow" + "plow" == "snowplow"
# => true

("snow" + "plow").equal?("snowplow")
# => false

Here, we end up with two String instances with the same content. Because they are distinct instances, #equal? returns false, and because their content is the same, #== returns true.

Do not implement #equal? in your own classes. It is not meant to be overridden. It’ll all end in tears.

Earlier in this article, I mentioned that #== has the property of reflexivity: an object is always equal to itself. Here is a related property for #equal?:

Property: Given objects a and b. If a.equal?(b), then a == b.

Ruby won’t automatically validate this property for your code. It’s up to you to ensure that this property holds when you implement the equality methods.

For example, recall the implementation of Employee#== from earlier in this article:

class Employee
  …
  def ==(other)
    super || (
      self.class == other.class &&
      !@id.nil? &&
      @id == other.id
    )
  end
end

The call to super on the first line makes this implementation of #== reflexive. This super invokes the default implementation of #==, which delegates to #equal?. Therefore, I could have used #equal?, rather than super:

class Employee
  …
  def ==(other)
    self.equal?(other) || (
      …
    )
  end
end

I prefer using super, though this is likely a matter of taste.

Hash equality

In Ruby, any object can be used as a key in a Hash. Strings, symbols, and numbers are commonly used as Hash keys, but instances of your own classes can function as Hash keys too — provided that you implement both #eql? and #hash.

The #eql? method

The #eql? method behaves similarly to #==:

"foo" == "foo"      # => true
"foo".eql?("foo")   # => true
"foo".eql?("bar")   # => false

However, #eql?, unlike #==, does not perform type coercion:

1 == 1.0        # => true
1.eql?(1.0)     # => false
1.0.eql?(1.0)   # => true

If #== doesn’t perform type coercion, the implementations of #eql? and #== will be identical. Rather than copy-pasting, however, we’ll put the implementation in #eql?, and let #== delegate to #eql?:

class Point
  …
  def ==(other)
    self.eql?(other)
  end
end

class Employee
  …
  def ==(other)
    self.eql?(other)
  end
end

I made the deliberate decision to put the implementation in #eql? and let #== delegate to it, rather than the other way around. If we were to let #eql? delegate to #==, there’s an increased risk that someone will update #== and inadvertently break the properties of #eql? (which I’ll mention below) in the process.

For the Path value object, whose #== method does perform type coercion, the implementation of #eql? will differ from the implementation of #==:

class Path
  …
  def ==(other)
    other.respond_to?(:to_str) &&
      to_str == other.to_str
  end

  def eql?(other)
    self.class == other.class &&
      @components == other.components
  end
end

Here, #== does not delegate to #eql?, nor the other way around.

A correct implementation of #eql? has the following two properties:

Property: Given objects a and b. If a.eql?(b), then a == b.

Property: Given objects a and b. If a.equal?(b), then a.eql?(b).

These two properties are not explicitly called out in the Ruby documentation. However, to the best of my knowledge, all implementations of #eql? and #== respect this property.

As before, Ruby will not automatically validate that these properties hold in your code. It’s up to you to ensure that these properties aren’t violated.

The #hash method

For an object to be usable as a key in a Hash, it needs to implement not only #eql?, but also #hash. This #hash method will return an integer, the hash code, that respects the following property:

Property: Given objects a and b. If a.eql?(b), then a.hash == b.hash.

Typically, the implementation of #hash creates an array of all attributes that constitute identity, and returns the hash of that array. For example, here is Point#hash:

class Point
  …
  def hash
    [self.class, @x, @y].hash
  end
end

For Path, the implementation of #hash will look similar:

class Path
  …
  def hash
    [self.class, @components].hash
  end
end

For the Employee class, which is an entity rather than a value object, the implementation of #hash will use the class and the @id:

class Employee
  …
  def hash
    [self.class, @id].hash
  end
end

If two objects are not equal, the hash code should ideally be different, too. This is not mandatory, however: it is okay for two non-equal objects to have the same hash code. Ruby will use #eql? to tell objects with identical hash codes apart.

Avoid XOR for calculating hash codes

A popular but problematic approach for implementing #hash uses XOR (the ^ operator). Such an implementation would calculate the hash codes of each individual attribute, and combine these hash codes with XOR. For example:

class Point
  …
  def hash
    # Do not do this!
    self.class.hash ^ @x.hash ^ @y.hash
  end
end

With such an implementation, the chance of a hash code collision, which means that multiple objects have the same hash code, is higher than with an implementation that delegates to Array#hash. Hash code collisions will degrade performance, and could potentially pose a denial-of-service security risk.

A better way, though still flawed, is to multiply the components of the hash code by unique prime numbers before combining them:

class Point
  …
  def hash
    # Do not do this!
    self.class.hash ^ (@x.hash * 3) ^ (@y.hash * 5)
  end
end

Such an implementation has additional performance overhead due to the new multiplication. It also requires mental effort to ensure the implementation is and remains correct.

An even better way of implementing #hash is the one I’ve laid out before — making use of Array#hash:

class Point
  …
  def hash
    [self.class, @x, @y].hash
  end
end

An implementation that uses Array#hash is simple, performs quite well, and produces hash codes with the lowest chance of collisions. It’s the best approach to implementing #hash.

Putting it together

With both #eql? and #hash in place, the Point, Path, and Employee objects can be used as hash keys:

points = {}
points[Point.new(11, 24)] = true

points[Point.new(11, 24)]   # => true
points[Point.new(10, 22)]   # => nil

Here, we use a Hash instance to keep track of a collection of Points. We can also use a Set for this, which uses a Hash under the hood, but provides a nicer API:

require "set"

points = Set.new
points << Point.new(11, 24)

points.include?(Point.new(11, 24))   # => true
points.include?(Point.new(10, 22))   # => false

Objects that are used in Sets need to have an implementation of both #eql? and #hash, just like objects that are used as hash keys.

Objects that perform type coercion, such as Path, can also be used as hash keys, and thus also in sets:

require "set"

home = Path.new("home", "denis")
also_home = Path.new("home", "denis")
elsewhere = Path.new("usr", "bin")

paths = Set.new
paths << home

paths.include?(home)        # => true
paths.include?(also_home)   # => true
paths.include?(elsewhere)   # => false

We now have an implementation of equality that works for all kinds of objects.

Mutability, nemesis of Equality

So far, the examples for value objects have assumed that these value objects are immutable. This is with good reason, because mutable value objects are far harder to deal with.

To illustrate this, consider a Point instance used as a hash key:

point = Point.new(5, 6)

collection = {}
collection[point] = 42

p point.hash               # => 421522
p collection.key?(point)   # => true

The problem arises when changing attributes of this point:

point.x = 10
p point.hash               # => 910895
p collection.key?(point)   # => false

Because the hash code is based on the attributes, and an attribute has changed, the hash code is no longer the same. As a result, collection no longer seems to contain the point. Uh oh!

There are no good ways to solve this problem, except for making value objects immutable.

This is not a problem with entities. This is because the #eql? and #hash methods of an entity are solely based on its explicit identity — not its attributes.

So far, we’ve covered #==, #eql?, and #hash. These three methods are sufficient for a correct implementation of equality. However, we can go further to improve that sweet Ruby developer experience, and implement #===.

Case equality (triple equals)

The #=== operator, also called the case equality operator, is not really an equality operator at all. Rather, it’s better to think of it as a membership testing operator. Consider the following:

10..15 === 14   # => true
80..99 === 14   # => false

Here, Range#=== checks whether a range covers a certain element. It is also common to use case expressions to achieve the same:

case 14
when 10..15
  puts "Kinda small!"
when 80..99
  puts "Kinda large!"
end

This is also where case equality gets its name. Triple-equals is called case equality, because case expressions use it.

You never need to use case. It’s possible to rewrite a case expression using if and ===. In general, case expressions tend to look cleaner. Compare:

if 10..15 === 14
  puts "Kinda small!"
elsif 80..99 === 14
  puts "Kinda large!"
end

The examples above all use Range#===, to check whether the range covers a certain number. Another commonly used implementation is Class#===, which checks whether an object is an instance of a class:

Integer === 15              # => true
Integer === 15.5            # => false

I’m rather fond of the #grep method, which uses #=== to select matching elements from an array. It can be shorter and sweeter than using #select:

[4, 2.0, 7, 6.1].grep(Integer)   # => [4, 7]
[4, 2.0, 7, 6.1].grep(2..6)      # => [4, 2.0]

# Same, but more verbose:
[4, 2.0, 7, 6.1].select { |num| Integer === num }
[4, 2.0, 7, 6.1].select { |num| 2..6 === num }

Regular expressions also implement #===. You can use it to check whether a string matches a regular expression:

phone = "+491573abcde"
case phone
when /00000/
  puts "Too many zeroes!"
when /[a-z]/
  puts "Your phone number has letters in it?!"
end

It helps to think of a regular expression as the (infinite) collection of all strings that can be produced by it. The set of all strings produced by /[a-z]/ includes the example string "+491573abcde". Similarly, you can think of a Class as the (infinite) collection of all its instances, and a Range as the collection of all elements in that range. This way of thinking clarifies that #=== really is a membership testing operator.

An example of a class that could implement #=== is a PathPattern class:

class PathPattern
  def initialize(string)
    @string = string
  end

  def ===(other)
    File.fnmatch(@string, other)
  end
end

An example instance is PathPattern.new("/bin/*"), which matches anything directly under the /bin directory, such as /bin/ruby, but not /var/log.

The implementation of PathPattern#=== uses Ruby’s built-in File.fnmatch to check whether the pattern string matches. Here is an example of it in use:

pattern = PathPattern.new("/bin/*")

pattern === "/bin/ruby"   # => true
pattern === "/var/log"    # => false

Worth noting is that File.fnmatch calls #to_str on its arguments. This way, #=== automatically works on other string-like objects as well, such as Path instances:

bin_ruby = Path.new("bin", "ruby")
var_log = Path.new("var", "log")

pattern = PathPattern.new("/bin/*")

pattern === bin_ruby       # => true
pattern === var_log        # => false

The PathPattern class implements #===, and therefore PathPattern instances work with case/when, too:

case "/home/denis"
when PathPattern.new("/home/*")
  puts "Home sweet home"
else
  puts "Somewhere, I guess"
end

Ordered comparison

For some objects, it’s useful not only to check whether two objects are the same, but how they are ordered. Are they larger? Smaller? Consider this Score class, which models the scoring system of my university in Ghent, Belgium.

class Score
  attr_reader :value

  def initialize(value)
    @value = value
  end

  def grade
    if @value < 10
      "failing"
    elsif @value < 14
      "passing"
    elsif @value < 16
      "distinction"
    elsif @value < 18
      "high distinction"
    else
      "highest distinction"
    end
  end
end

(I was a terrible student. I’m not sure if this was really how the scoring even worked — but as an example, it will do just fine.)

In any case, we benefit from having such a Score class. We can encode relevant logic on there, such as determining the grade, and checking whether or not a score is passing. For example, it might be useful to get the lowest and highest score out of a list:

scores = [
  Score.new(6),
  Score.new(17),
  Score.new(14),
  Score.new(13),
  Score.new(11),
]

p scores.min
p scores.max

However, as it stands right now, the expressions scores.min and scores.max will result in an error: comparison of Score with Score failed (ArgumentError). We haven’t told Ruby how to compare two Score objects. We can do so by implementing Score#<=>:

class Score
  …
  def <=>(other)
    return nil unless other.class == self.class

    @value <=> other.value
  end
end

An implementation of #<=> returns four possible values:

It returns 0 when the two objects are equal.
It returns -1 when self is less than other.
It returns 1 when self is greater than other.
It returns nil when the two objects cannot be compared.

The #<=> and #== operators are connected:

Property: Given objects a and b. If (a <=> b) == 0, then a == b.

Property: Given objects a and b. If (a <=> b) != 0, then a != b.

As before, it’s up to you to ensure that these properties hold when implementing #== and #<=>. Ruby won’t check this for you.

For simplicity, I’ve left out the implementation Score#== in the Score example above. It’d certainly be good to have that, though.

In the case of Score#<=>, we bail out if other is not a score, and otherwise, we call #<=> on the two values. We can check that this works: the expression Score.new(6) <=> Score.new(12) evaluates to -1, which is correct because a score of 6 is lower than a score of 12.³

With Score#<=> in place, scores.max now returns the maximum score. Other methods such as #min, #minmax, and #sort work as well.

However, we can’t yet use operators like <. The expression scores[0] < scores[1], for example, will raise an undefined method error: undefined method `<' for #<Score:0x00112233 @value=6>. We can solve that by including the Comparable mixin:

class Score
  include Comparable

  …
  def <=>(other)
    …
  end
end

By including Comparable, the Score class automatically gains the <, <=, >, and >= operators, which all call <=> internally. The expression scores[0] < scores[1] now evaluates to a boolean, as expected.

The Comparable mixin also provides other useful methods such as #between? and #clamp.

Wrapping up

We talked about the following topics:

the #== operator, used for basic equality, with optional type coercion.
#equal?, which checks whether two objects are the same instance.
#eql? and #hash, which are used for testing whether an object is a key in a hash.
#===, which isn’t quite an equality operator, but rather a “is kind of” or “is member of” operator.
#<=> for ordered comparison, along with the Comparable module, which provides operators such as < and >=.

You now know all you need to know about implementing equality in Ruby.

Using a modal lexer for parsing sublanguages

2022-03-19T23:00:00Z

How do you write a lexer for print "one plus two is ${1+2}";? The string interpolation part makes this tricky, as the string forms a sublanguage that is different from the main language. To make things worse, interpolation can be nested. Textbook lexers won’t cut it, and we need something stronger.

Enter modal lexers. These lexers can switch between different modes for lexing different languages. In this article, I’ll run you through an example of how to build such a modal lexer.

For this article, I’m using Ruby as the programming language. The same technique will work in other programming languages too. No external dependencies (gems) are used.

Ruby’s string scanner
A basic lexer
1. Testing it out
A modal lexer
Hints for the parser
Wrapping up
References

Ruby’s string scanner

Ruby provides a StringScanner class, which is is an excellent building block for lexers. It offers a clean API to split up a string into individual parts.

I’ll run you through a simple example, so that you can get a feel of how it works. First, we require strscan, and then we instantiate the StringScanner with the string to scan:

require "strscan"

str = "let amount = 5;"
scanner = StringScanner.new(str)

The #scan method, when given a string part, will attempt to match this given string part:

scanner.scan("let") # => "let"

The scanner starts at the beginning of the string. When a match is successful, the scanner will move forward. Because we just matched "let", the scanner has moved past this "let", and attempting to match it again will fail:

scanner.scan("let") # => nil

The #scan method returns the matched string part, or nil if there is no match. In the latter case, the scanner stays at its current position.

This method can also take a regular expression:

scanner.scan(/\s+/) # => " "

The regular expression \s+ matches one or more whitespace characters, such as spaces, tabs, and newlines.

Now that we’ve matched "let" and a space character, we can match the next part: an identifier, consisting of alphanumeric characters. For this, the regular expression \w+ comes in handy, which matches one or more alphanumeric characters:

scanner.scan(/\w+/) # => "amount"

Another useful method is #matched, which returns the matched string part:

scanner.matched # => "amount"

The StringScanner provides more functionality, but #scan and #matched is all we need to write a lexer.

A basic lexer

Before writing a modal lexer, we’ll start with a basic lexer:

class BasicLexer
  def initialize(str)
    @scanner = StringScanner.new(str)
  end

This BasicLexer is constructed with a string. The string itself isn’t important; we’ll only use the StringScanner.

  def next_token

The only public method for this lexer is #next_token, which will return a single token, or nil when there are no more tokens.

    if @scanner.eos?
      nil

If the scanner is at the end of the string (eos stands for end-of-string), we’ll return nil.

    elsif @scanner.scan(/\s+/)
      [:WHITESPACE, nil]

If we match one or more whitespace characters (\s+), we return a WHITESPACE token.

Tokens are arrays with two elements: first, the token type as an uppercase symbol, and second, an optional value. The value for this WHITESPACE token is nil; we won’t use the value for anything.

    elsif @scanner.scan("+")
      [:PLUS, nil]
    elsif @scanner.scan("*")
      [:TIMES, nil]
    elsif @scanner.scan("-")
      [:MINUS, nil]
    elsif @scanner.scan("/")
      [:DIVIDE, nil]
    elsif @scanner.scan(";")
      [:SEMICOLON, nil]
    elsif @scanner.scan("=")
      [:EQUAL, nil]

Here, we match against some single-character syntax elements.

    elsif @scanner.scan(/\d+/)
      [:NUMBER, @scanner.matched]

We match against the regular expression \d+, which matches one or more digits. For this token, we record the value: @scanner.matched is the string containing the digits, e.g. "42".

The next bit deals with keywords and identifiers:

    elsif @scanner.scan(/\w+/)
      case @scanner.matched
      when "let"
        [:KW_LET, nil]
      when "print"
        [:KW_PRINT, nil]
      else
        [:IDENTIFIER, @scanner.matched]
      end

This code reads a sequence of alphanumeric characters — anything that could be a keyword or an identifier. Then, we check whether the matched string part is one of the predefined keywords.

Note that this correctly handles identifiers that start with a keyword. For example, "letter" is an identifier, not the keyword "let" followed by the identifier "ter".

    else
      char = @scanner.getch
      raise "Unknown character: #{char}"
    end
  end
end

Lastly, in case the scanner did not match anything, we raise an exception. It’s a crude way of reporting errors, but it’ll do for this basic lexer.

Testing it out

To test out this basic lexer, construct it with a string, and keep calling #next_token until that method returns nil:

str = "let amount = 42;"
lexer = BasicLexer.new(str)

loop do
  token = lexer.next_token
  break unless token

  p token
end

This code spits out the following, as expected:

[:KW_LET, nil]
[:WHITESPACE, nil]
[:IDENTIFIER, "amount"]
[:WHITESPACE, nil]
[:EQUAL, nil]
[:WHITESPACE, nil]
[:NUMBER, "42"]
[:SEMICOLON, nil]

We want to be able to parse code like this:

print "one plus two is ${1+2}";

In this hypothetical programming language, the expression between ${ and } is evaluated and embedded in the string. This code would thus effectively be the same as print "one plus two is 3";.

The contents of the string are in a different language. Outside of a string, ${ would be a lexer error. Also, one and plus are string literal parts, not identifiers. To make our lexer support this code, we’ll allow our lexer two switch between two modes.¹ First, we’ll have a main lexer mode, which will do pretty much the same as earlier BasicLexer. Secondly, we’ll have a string lexer mode, which will be used for handling the contents of strings.

However, toggling between these two modes is not enough. It is possible to have string interpolation nested within string interpolation:

print "why ${"${2+6}" * 5}";

The easiest way to support this in our ModalLexer class is using a stack of lexer modes:

class ModalLexer
  def initialize(str)
    scanner = StringScanner.new(str)
    mode = MainLexerMode.new(scanner)
    @mode_stack = [mode]
  end

The ModalLexer class will, like the previous BasicLexer class, be constructed with a string. It creates a stack of lexer modes, which will contain just a MainLexerMode to begin with. This string scanner will be shared across lexer modes.

  def next_token
    mode = @mode_stack.last
    mode.next_token(@mode_stack)
  end
end

Just like with BasicLexer, the only public method for this lexer is #next_token. However, the implementation here delegates to the current lexer mode, which sits at the top of the stack.

The #next_token method for lexer modes takes the mode stack as an argument. This will allow the lexer modes to modify this stack.

The main lexer mode

The implementation of MainLexerMode is similar to that of BasicLexer:

class MainLexerMode
  def initialize(scanner)
    @scanner = scanner
  end

A MainLexerMode is constructed with a string scanner, rather than a string. The string scanner needs to be shared across lexer modes. This way, when switching lexer modes, the new lexer mode can continue where the previous one left off.

  def next_token(mode_stack)

The #next_token method takes the stack of lexer modes as a parameter.

    if @scanner.eos?
      nil

As with BasicLexer, if the scanner is at the end of the string, we return nil.

    elsif @scanner.scan('"')
      mode = StringLexerMode.new(@scanner)
      mode_stack.push(mode)
      [:STRING_START, nil]

When we find a " character, we push a new StringLexerMode onto the lexer mode stack, and return a STRING_START token. This way, the next call to #next_token will be delegated to this new StringLexerMode instance, at the top of the stack.

The rest of the MainLexerMode implementation is identical to BasicLexer.

The string lexer mode

The StringLexerMode class is the lexer mode that is used inside string literals.

class StringLexerMode
  def initialize(scanner)
    @scanner = scanner
  end

As with MainLexerMode, a StringLexerMode is constructed with a string scanner.

  def next_token(mode_stack)
    if @scanner.eos?
      nil

The #next_token method starts identical to MainLexerMode, returning nil when the scanner is at the end of the string.

    elsif @scanner.scan('"')
      mode_stack.pop
      [:STRING_END, nil]

The " character indicates the end of the string. We pop the current lexer mode off the stack, and return a STRING_END token.

    elsif @scanner.scan("${")
      mode = InterpolationLexerMode.new(@scanner)
      mode_stack.push(mode)
      [:STRING_INTERP_START, nil]

When we find ${, we push a new lexer mode: InterpolationLexerMode. This is a to-be-implemented lexer mode which will be similar to MainLexerMode. We return a STRING_INTERP_START token.

    elsif @scanner.scan("$")
      [:STRING_PART_LIT, @scanner.matched]

When we find a $ character, we return a STRING_PART_LIT token with that $ character as the content of the token. A STRING_PART_LIT token is used for literal string content. We know that this $ won’t be followed by a {, because otherwise it’d have matched the earlier ${ case.

    elsif @scanner.scan(/[^"$]+/)
      [:STRING_PART_LIT, @scanner.matched]

When we find a character sequence that includes neither " nor $, we return a STRING_PART_LIT token.

    else
      char = @scanner.getch
      raise "Unknown character: #{char}"
    end
  end
end

Lastly, if the scanner did not match anything, we raise an exception.

The string interpolation lexer mode

The string interpolation lexer mode is used between ${ and } inside a string. This lexer mode is identical to MainLexerMode, with the difference that it needs to pop the lexer mode stack when it finds }.

class InterpolationLexerMode
  def initialize(scanner)
    @scanner = scanner
    @delegate = MainLexerMode.new(@scanner)
  end

The InterpolationLexerMode is constructed with a scanner, similar to the other lexer modes. It also creates a MainLexerMode, to which it will delegate.

  def next_token(mode_stack)
    if @scanner.scan("}")
      mode_stack.pop
      [:STRING_INTERP_END, nil]

If we find a } character, we pop the lexer mode stack. Afterwards, the lexer mode at the top of the stack will be a StringLexerMode. We return a STRING_INTERP_END token.

    else
      @delegate.next_token(mode_stack)
    end
  end
end

Otherwise, we delegate the #next_token call to the MainLexerMode instance.

Testing it out

As before, we can test out this modal lexer, by construct it with a string, and keep calling #next_token until that method returns nil:

string = 'print "one plus two is ${1+2}.";'
lexer = ModalLexer.new(string)

loop do
  token = lexer.next_token
  break unless token

  p token
end

This code spits out the following:

[:KW_PRINT, nil]
[:WHITESPACE, nil]
[:STRING_START, nil]
[:STRING_PART_LIT, "one plus two is "]
[:STRING_INTERP_START, nil]
[:NUMBER, "1"]
[:PLUS, nil]
[:NUMBER, "2"]
[:STRING_INTERP_END, nil]
[:STRING_PART_LIT, "."]
[:STRING_END, nil]
[:SEMICOLON, nil]

Note how the string starts with STRING_START and ends with STRING_END. Inside, we have literal parts (STRING_PART_LIT), as well as interpolated parts, which start with STRING_INTERP_START and end with STRING_INTERP_END.

Now everything is in place for writing a parser.

Hints for the parser

Writing a parser isn’t quite in scope for this article, but it wouldn’t be complete without touching on the subject at least briefly.

In a parser, you’d define a string as the collection of parts between a STRING_START and STRING_END token:

string      -> STRING_START string_part* STRING_END

A string part can be a literal part, or an interpolated part.

string_part -> STRING_PART_LIT

string_part -> STRING_INTERP_START
               expression
               STRING_INTERP_END

For string parts that are STRING_PART_LIT, you’d extract the token value. For interpolated string parts, you’d evaluate the expression and convert it to a string. This way, parsing simple strings works:

print "ab5yz";

(print
  (string
    (lit "ab5yz")))

Parsing strings with interpolation works as well:

print "ab${2+3}yz";

(print
  (string
    (lit "ab")
    (expr
      (+ (num 2) (num 3)))
    (lit "yz")))

To simplify this AST, you might consider de-sugaring this syntax. Here’s an example where I’ve replaced the string and expr AST nodes with calls to my concat() and toString() functions:

(print
  (funcall
    "concat"
    (lit "ab")
    (funcall
      "toString"
      (+ (num 2) (num 3)))
    (lit "yz")))

Wrapping up

This wraps up the modal lexing example, though plenty can can be done to improve this lexer. These are the bits I left out, for the sake of simplicity:

The StringLexerMode does not handle escape sequences, like \", \\, or \$. Escape sequences are indispensable in a real-world string lexer.
The lexer does not track source location (line number, column number, or filename). This too is indispensable for reporting errors.
Speaking of errors: The lexer’s error handling leaves much to be desired.
The lexer emits WHITESPACE tokens, which might not be desirable. These could be handled by the parser, or stripped entirely beforehand.
I’ve not paid attention to performance characteristics of this lexer. My assumption is that StringScanner is fast enough, especially with pre-compiled regular expressions. However, it might be worth looking into other approaches and tools for lexing.
The InterpolationLexerMode will pop the lexer mode stack as soon as it finds a }. This will cause problems if the } token is used for other purposes as well, such as defining functions (e.g. fn a() { … }) or creating maps (e.g. let thing = { value: 123 };). The trick here is to push a MainLexerMode when finding a {, and letting MainLexerMode pop the stack when finding a }. Now InterpolationLexerMode is redundant and can be substituted with MainLexerMode.

This modal lexing technique is not just usable for handling string interpolation. It can handle nested comment as well (e.g. /* left /* inner comment */ right */), and it even allows embedding entire different languages.

The possibilities are limited only by your imagination.

References

This article was inspired by How OSH Uses Lexer Modes (2016), which introduces the concept of modal lexing. You might also be interested in these:

This change turns the lexer from a finite automaton into a pushdown automaton. This means that the lexer no longer recognizes a regular language but a context-free language. The textbook theory says that lexers should be regular. But modal lexers are such an elegant solution that I’m quite happy to drop the theory and go with what actually works. ↩

Drive change by writing structured proposal documents

2021-07-16T22:00:00Z

In any collaborative environment, opportunities for change often arise. When such an opportunity presents itself, write up a structured proposal document describing how to tackle this opportunity. With this document, a decision on whether or not to implement the proposal can be made quickly and confidently.

I use this approach extensively, both at my employers as well as for side projects that are used by others. These documents are often called RFCs (short for requests for comment).

Motivation

Whether when working alone or collaborating with others, opportunities for change will come up. While many such changes are trivial, some are not:

Changes that impact others, such as restructuring the filesystem of a project, or reformatting all source code to the latest standards.
Changes that are not easily reversible, such as the choice of programming language or framework to use for a new project, or the switch to a new paid subscription of a third-party tool.

To make decisions on how to move forward with such opportunities for change, we need to have as much information as possible about the change and its implications.

We want to collect all relevant information regarding a potential decision from all stakeholders. At the same time, we want to make sure that all stakeholders understand the context and motivation around this change, and the impact it will have.

Proposed solution

When an opportunity for change arises, write up a document that describes how to take advantage of this opportunity. This proposal document has the following format:

Context (optional): What background information (if any) is needed to understand the rest of this document? Consider adding a glossary in this section to clarify any jargon that readers might not be familiar with.
Motivation: What pain point does this proposal address?
Solution constraints (optional): What is the set of requirements for every solution for the aforementioned pain point? Consider organising these in “must have”, “should have”, “nice to have”, and “optional”.
Proposed solution: How will the aforementioned pain points be addressed? What are the implications of this proposed solution? What are the advantages and disadvantages of this solution? Consider adding examples of how this solution would work.
Alternative solutions (optional): What other solutions would also alleviate this pain point? What advantages and disadvantages do they have compared to the proposed solution?
Open questions and concerns (optional): Are there any questions that have not (yet) been answered? Are there concerns that have not (yet) been taken into account? In this section, surface all questions and concerns, even if they might not need to be tackled in this document.

Write this document on a platform that supports collaboration, and has fine-grained commenting functionality, such as Google Docs or Notion. While other approaches work (e.g. sending Word documents and updates to them over email), they have long and cumbersome feedback loops. A collaborative platform tightens the feedback loop, and thereby increases participation. Increased participation means more information is shared, and thus increases the quality of the decisions.

Perform research for decisions asynchronously. Avoid making decisions synchronously, e.g. in a real-time meeting.

Ownership

Ensure that a single person is assigned as the owner of this document. The owner is responsible for writing and maintaining this document. All suggested changes to the document should go through the document’s owner.¹

Lifecycle

Until the proposed solution in this document is accepted, keep the proposal document updated with any new information that comes in. Ensure that all people involved are aware of updates to the document.

Ensure that there is a clear point where the proposal is either accepted or rejected. Setting a deadline up front can be useful.

Example

This very document you’re reading is structured using this format.

Advantages

By documenting the decisions around proposals, a historical record can be kept, and can be referred to later on if needed. In particular, it can shed clarity on the constraints that were (and perhaps no longer are) present when a past decision was made.

This approach ensures all potential stakeholders are involved in the process, and that their input is taken into account.

With this approach, not everyone is required to participate in the process. In this way, this approach can be used to get people on board with breaking changes by radiating intent.

Disadvantages

This approach is not well-suited for trivial decisions, nor for urgent decisions.

Open questions and concerns

It might be worth adding a RACI matrix to the document, to state who is responsible, accountable, consulted, and informed.
It might be worth documenting the thought process that went into the proposed solution. References (bibliography) would then probably go into a subsection of the Proposed solution section.

If “owner” sounds too rigid, consider “steward” instead. ↩

Budget for replacing past purchases

2021-06-12T22:00:00Z

Unexpected expenses can cause financial anxiety. This is true for me, even though I have a good salary and a solid emergency buffer. One way in which I reduce my financial anxiety is by predicting expenses based on the expected lifetime of a purchase.

Limited-lifetime items with non-trivial cost

Some items have a limited lifetime (e.g. 4–5 years) and have a non-trivial cost (e.g. €1500). A non-trivial cost is a cost that, if it were to suddenly show up, would stir up my budget enough to cause stress. For me, these items fit primarily into three categories:¹

Electronic devices: laptop, phone, speakers, external display, …
Larger appliances: dishwasher, washing machine, …
Furniture with limited lifetime: office chair, mattress, …

This explicitly excludes items with non-trivial cost that have a long (expected) lifetime. For example, this does this not include my expensive (but gorgeous) kitchen table, which hopefully will last forever.

How to budget for replacement

Starting out with this technique requires some initial setup. Once that’s done, the technique is easy to apply on a monthly basis.

Starting out

Make a list of all items with limited lifetime and non-trivial cost. You can use the categorization above as a guideline.

For each item, roughly estimate its remaining expected lifetime, and the cost to replace it. For example: I bought my MacBook Pro in late 2017 and expect its lifetime to be 5 years. At the time of writing (early 2021), the expected remaining lifetime is roughly 20 months, and replacing it would cost roughly €1500.

Monthly budgeting

Each month, budget the amount of money needed to cover the replacement cost in the expected lifetime. For example, the €1500 for replacing my MacBook Pro over the remaining 20 months of its lifetime comes down to €75/month.

For each future purchase with limited lifetime and non-trivial cost, do this budgeting calculation right when you make the purchase. For example, if I were to replace my MacBook Pro right now (assuming the replacement has the same cost of €1500 and the same expected lifetime of 5 years), I’d immediately start budgeting €1500 spread over 5 years, which comes down to €25/month.²

Lifetime vs. warranty

It is safe to use the warranty period as the lifetime, but this can be overly pessimistic. For example, my 2017 MacBook Pro is out of warranty, but I expect to keep on using it for at least another year.

In the end, it comes down to how much risk (uncertainty) you’re willing to take on.

Advantages

I am able to deal with unanticipated expenses in a straightforward, no-anxiety way. Ideally, I only replace the item once I have all the money for replacement saved up, but even if I need to replace it before its expected lifetime ends, I will at least have a helpful part of the money saved up.
I see which items have extended past their expected lifetime. Because these items have been fully-funded, I can replace them with no financial impact — if I need or want to.
It makes me prefer purchasing long-lasting items over short-lasting ones, as short-lasting items will have a potentially higher monthly cost (for budgeting) associated with them.

Disadvantages

It looks rather gloomy to immediately start budgeting for replacing an expensive item, right after purchasing it.

This categorization isn’t all-encompassing: some items with non-trivial cost and limited lifetime don’t fit in properly, e.g. my glasses. ↩
Budgeting software exists to make this easier! I use You Need A Budget, which works nicely with this approach. I’m also keeping my eye on Actual. ↩

Horizontally scrolling multi-column layout: a retrospective

2021-06-08T22:00:00Z

On wide screens, and especially ultra-wide screens, a single column of text often requires a considerable amount of whitespace to the left and the right. This whitespace is needed to prevent the text lines in the column from growing overly long, and thus hard to read.

All this whitespace is wasted screen real estate. In this experiment, I aimed at using screen real estate more efficiently using CSS’ multi-column support.

See it in action.

What worked

So much less wasted screen real estate!

Reading an article involved a lot less scrolling up and down to refer to previous bits of in the article. This makes sense, as there’s considerably more text visible on the screen.

I’m fond of the aesthetic: it reminds me of a book or a magazine.

What didn’t work

As excited as I was, this experiment revealed some issues.

Controlling column widths

CSS’ multi-column support really wants to divide the available space in an integer number of columns. This means that the actual width of the columns depends on the width of the container (in this experiment, the container is the window). This happens even when setting an explicit column width using the column-width property, because this property defines the ideal column width, not the actual column width. Quoting MDN:

The column-width CSS property sets the ideal column width in a multi-column layout. The container will have as many columns as can fit without any of them having a width less than the column-width value.

As a consequence, it becomes non-obvious that you can scroll right: there is no indication that there are additional columns past the last-visible column. To resolve this problem in this experiment, I cheated and made the next column half visible, which I think gives enough of a hint that there is more content beyond what is visible.

Furthermore, as a result of column widths being calculated based on the container width, resizing the window will resize the columns as well. In this experiment, this behavior is jarring, as text reflows, especially when the viewport is already scrolled considerably to the right.

Getting scrolling to work

Horizontal scrolling is easy using a trackpad, but I had to add some custom JavaScript to make the mouse wheel scroll horizontally as well. I intercepted the page-up and page-down keys as well, and injected some custom JavaScript to make smooth scrolling work.

Unfortunately, the custom JavaScript is not doing its job quite well. For example, holding space or page-down will not make the scrolling go faster, but instead it will confusingly slow it down. This is a bug in the JavaScript code, but I’d rather rely on native scrolling than on JavaScript.

I saw mentions of a JavaScript-less solution that involves multiple nested CSS transformations (rotating the container 90° and then rotating the container’s children the opposite way), but I could not get this approach to work.

Padding

For reasons I don’t fully understand, ensuring that the last column has some right padding is particularly difficult. Without this right padding, the last column sits awkwardly against the window border. I got this to work by adding a fat right border to every element inside the multi-column container, which is a terrible hack and I feel bad about it. Someone needs to come up with a better idea.

Controlling column breaks

In multi-column media, like in multi-page media, it becomes important to control how elements are broken across columns. The break-inside: avoid CSS property works well, though it seems to be the only well-supported property. CSS’ widows and orphans support is spotty across browsers. So is the break-after: avoid property, which I’d love to have used on headers — keeping headers together with the first-following paragraph makes a difference.

Open questions and future work

Is horizontal scrolling a good idea? It’s unconventional, but does that mean it should be avoided?

Overflowing columns in the block direction would be sweet. Quoting Rachel Andrew:

For Level 2 of the specification, we are considering how to enable a method by which overflow columns, those which currently end up causing the horizontal scrollbar, could instead be created in the block direction. This would mean that you could have a multicol container with a height, and once the content had made columns which filled that container, a new set of columns would be created below.

(Emphasis mine.) This approach would eliminate issues with scrolling (no JavaScript needed, and scrolling would be vertical and thus more conventional).

Closing thoughts

CSS’ multi-column support is not usable for long-form content yet. While there is certainly progress made, and the opportunities are exciting, the usability problems related to non-conventional scrolling make it difficult to use, and browser support is too spotty to provide a reliable, comfortable experience.

Using Lenses to build complex React forms in a type-safe way

2020-11-23T23:00:00Z

Forms in React are not straightforward to build. For trivial forms, plain-old React is good, but once the forms get more complex, you’ll likely find yourself searching for a good form-building library for React.

In this article, I propose an alternative approach to building forms, one based on lenses, tightly integrated with TypeScript¹ to tighten the feedback loop, reducing the change of mistakes and speeding up development.

The trouble with forms
1. Tightening the feedback loop using type safety
2. Demo app
What is a lens?
1. Lenses, more formally
2. Conveniently creating lenses with forProp()
Lenses for forms
1. Minimal form example
2. Prettier form example
Handling nested data by composing lenses
Handling lists
1. Adding and removing list elements
Improving on single-select dropdowns
1. Dropdowns versus radio buttons
Future work
Closing thoughts

The trouble with forms

During my time at BCG Digital Ventures, I worked on an internal tool where one particular page has a remarkably complex form: multiple dozens of text fields, radio buttons, checkboxes, and dropdowns, and in addition to all that, it had editable lists inside editable lists, with even more form fields inside.

The implementation of this form had gone through several revisions to increase its robustness and make it more maintainable. The penultimate version was built with Formik, which removed a ton of complexity and provided a good out-of-the-box experience.

However, even the version built with Formik had issues. Like other mainstream React form-handling libraries, Formik has trouble dealing with highly-complex nested data structures. In particular, I found that spelling mistakes in field names² create subtle bugs that are hard to detect, and require extensive automated or manual testing to uncover.

We can improve on this by making good use of the type system!

Tightening the feedback loop using type safety

Many mainstream React form libraries are written in plain-old JavaScript. Some of them are written in TypeScript, but are lax with types, and provide little type safety. Type safety is useful because tightens the feedback loop: misspelled field names will be highlighted as errors in the IDE in real time, and the IDE will provide reliable and useful autocompletion with integrated documentation.

The functional programming community came up with the concept of lenses. Lenses, as you’ll see in more detail further down, are in essence bi-directional accessors for immutable objects. The immutability fits in nicely with React, and the type safety they provide tightens the development feedback loop.

I reimplemented the remarkably complex form using lenses, and was delighted with the outcome: lenses are capable of handling highly-complex data in a type-safe way. Lenses helped eliminated bugs and prevent regressions, and significantly sped up development. Lenses allow building complex forms quickly and with confidence.

Demo app

Below, you will find a live demo of something you can build with lenses. You can find the full implementation in the matching demo Git repository.

What is a lens?

To describe what a lens is, we’ll use an example to create a lens from scratch.

Imagine a Person type, and an instance of Person that represents Sherlock Holmes:

type Person = {
  firstName: string;
  lastName: string;
};

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes"
};

We can get Sherlock’s first name:

sherlock.firstName
  // -> "Sherlock"

The game is afoot!

We could also write some code to update the first name, which we’ll do in a purely functional way, meaning that we won’t modify the object itself, but rather return a new object with the updated data:

let locky = { ...sherlock, firstName: "Locky" }
  // -> {
  //      firstName: "Locky",
  //      lastName: "Holmes"
  //    }

Perhaps Sherlock wouldn’t like to be called Locky. We’ll never know.

We can create functions for getting and updating a person’s first name:

let getFirstName =
  (person: Person) => person.firstName

let setFirstName =
  (person: Person, firstName: string) =>
    ({ ...person, firstName: firstName })

Let’s fix Sherlock’s name:

getFirstName(locky)
  // -> "Locky"

setFirstName(locky, "Sherlock")
  // -> {
  //      firstName: "Sherlock",
  //      lastName: "Holmes"
  //    }

We can combine the getter and the setter into a single object:

let firstNameLens = {
  get: (person: Person) =>
    person.firstName

  set: (person: Person, firstName: string) =>
    ({ ...person, firstName: firstName })
}

firstNameLens.get(locky)
  // -> "Locky"

firstNameLens.set(locky, "Sherlock")
  // -> {
  //      firstName: "Sherlock",
  //      lastName: "Holmes"
  //    }

Congratulations: firstNameLens is your first lens!

Lenses, more formally

The lens that we constructed above has the following type:

type PersonFirstNameLens = {
  // Given a person,
  // get a string (the first name)
  get: (person: Person) => string;

  // Given a person and a string
  // (the first name),
  // return a new person
  set: (person: Person, firstName: string) => Person;
};

This lens is for two specific types:

The Person type is the top type: the type that contains the data that you want to extract (using get), or the data that you want to update (using set).
The string type is the focused type: the type of the extracted data.

With these two types in mind, we can construct a generic Lens type, with two type parameters (T for the top type, and F for the focused type):

type Lens<T, F> = {
  get: (t: T) => F;
  set: (t: T, f: F) => T;
};

The type definition makes it clear: a lens is the combination of a getter and a setter, for an immutable data structure.

Conveniently creating lenses with forProp()

It is convenient to have a function that can automatically create a lens for a property. This is where forProp() comes in:³

let firstNameLens =
  forProp<Person>()("firstName");

A lens returned by forProp() behaves exactly the same as a manually-constructed one:

let john: Person = {
  firstName: "John",
  lastName: "Watson"
};

firstNameLens.get(john);
  // -> "John"
firstNameLens.set(john, "Joe");
  // -> { firstName: "Joe", lastName: "Watson" }

I’ll make a guess that John Watson wouldn’t like to be called Joe.

The forProp() function is type-safe, as it won’t accept non-existent properties:

// Type error!
// Person has no property middleName
let middleNameLens =
  forProp<Person>()("middleName");

We’ll not talk about the implementation, but you can check out its implementation in the demo sandbox.

Lenses for forms

A lens-powered form field needs three properties:

interface BareTextFieldProps<T> {
  lens: Lens<T, string>;
  top: T;
  setTop: (t: T) => void;
}

A form field needs the lens (for getting and setting the data for this field), but also top and setTop(), which are used for getting and setting the top-level object.

Note the similarity between top and setTop() and what the React useState hook returns — we’ll come back to this later.

This minimalist text field’s implementation is as follows:

export let BareTextField = <T extends any>({
  lens,
  top,
  setTop
}: BareTextFieldProps<T>) => {
  // Read value through lens
  let value = lens.get(top);

  // Replace top with new instance
  // updated through lens
  let setValue = (newValue: string) => {
    setTop(lens.set(top, newValue));
  };

  return (
    <input
      type="text"
      value={value}
      onChange={e => setValue(e.target.value)}
    />
  );
};

This React component is a controlled component, so the wrapped <input> component is given both a value prop and an onChange prop.

Minimal form example

We’ll create a form for a new person. First, we’ll need our lenses:

let firstNameLens =
  forProp<Person>()("firstName");

let lastNameLens =
  forProp<Person>()("lastName");

We’ll also need a way to create a blank person object:

let newPerson = (): Person => ({
  firstName: "",
  lastName: ""
});

The skeleton of our form will look like this:

let PersonForm = () => {
  let [person, setPerson] = useState(newPerson);

  return (
    <>
      {/* to do: add form fields here */}
      <pre>{JSON.stringify(person, null, 2)}</pre>
    </>
  );
};

When the form is created, the person variable is initialized to a new person.

At the end of the form, we show the pretty-printed representation of the person, so that you can see that it indeed is updating the person properly.

Let’s add the fields for the first name and last name:

<>
  <BareTextField
    top={person}
    setTop={setPerson}
    lens={firstNameLens}
  />

  <BareTextField
    top={person}
    setTop={setPerson}
    lens={lastNameLens}
  />

  <pre>{JSON.stringify(person, null, 2)}</pre>
</>

We can reduce the amount of boilerplate by creating an object f that contains top and setTop(), so that we can pass it to the text fields succinctly:

let PersonForm = () => {
  let [person, setPerson] = useState(newPerson);
  let f = { top: person, setTop: setPerson };

  return (
    <>
      <BareTextField {...f} lens={firstNameLens} />
      <BareTextField {...f} lens={lastNameLens} />
      <pre>{JSON.stringify(person, null, 2)}</pre>
    </>
  );
};

With this approach, you can build forms with nested objects in a terse and type-safe way.

Prettier form example

The text field we’ve created so far is nothing but a wrapper for an input element. We can build a more full-fledged text field by adding a label and some styling (I am partial to utility-first CSS):

interface TextFieldProps<T> extends BareTextFieldProps<T> {
  label: string;
}

let TextField = <T extends any>({
  lens,
  top,
  setTop,
  label
}: TextFieldProps<T>) => (
  <label className="block pb-6">
    <div className="font-bold">{label}</div>
    <BareTextField
      lens={lens}
      top={top}
      setTop={setTop}
    />
  </label>
);

Once we replace our BareTextField usage in the form with TextField (now with label), we get a nicer form:

<>
  <TextField
    {...f}
    lens={firstNameLens}
    label="First name"
  />
  <TextField
    {...f}
    lens={lastNameLens}
    label="Last name"
  />
  <pre>{JSON.stringify(person, null, 2)}</pre>
</>

Handling nested data by composing lenses

We are able to build forms for simple objects now, but not for nested objects. Let’s fix that.

Image a Person type with an address inside it:

type Address = {
  street: string;
  number: string;
};

type Person = {
  firstName: string;
  lastName: string;
  address: Address;
};

Let’s take Sherlock as an example person:

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes",
  address: {
    street: "Butcher Street",
    number: "221b"
  }
}

We can get Sherlock’s street easily:

sherlock.address.street

Updating Sherlock’s street is more cumbersome without lenses:

let sherlock2 = {
  ...sherlock,
  address: {
    ...sherlock.address,
    street: "Baker Street"
  }
}

If Address were a standalone type, we’d be able to update it succinctly with a lens:

let sherlocksHome: Address = {
  street: "Butcher Street",
  number: "221b"
}

let streetLens =
  forProp<Address>()("street");

streetLens.set(sherlocksHome, "Baker Street");
  // -> { street: "Baker Street", number: "221b" }

We can, however, create a lens for a person’s address, and for an address’ street, and compose them, so that we get a lens for a person’s street:

let addressLens =
  forProp<Person>()("address");

let streetLens =
  forProp<Address>()("street");

let addressStreetLens =
  compose(addressLens, streetLens);

The addressStreetLens lens “drills down” into the person type. It behaves like any other lens:

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes",
  address: {
    street: "Butcher Street",
    number: "221b"
  }
}

addressStreetLens.set(sherlock, "Baker Street");
  // -> {
  //      firstName: "Sherlock",
  //      lastName: "Holmes",
  //      address: {
  //        street: "Baker Street",
  //        number: "221b"
  //      }
  //    }

This works for forms too, like any other lens. Let’s create the lenses we need first:

let addressLens =
  forProp<Person>()("address")
let streetLens =
  forProp<Address>()("street");
let houseNumberLens =
  forProp<Address>()("number");
let addressStreetLens =
  compose(addressLens, streetLens);
let addressNumberLens =
  compose(addressLens, houseNumberLens);

Now we can use them in a form:

<>
  <TextField
    {...f}
    lens={firstNameLens}
    label="First name" />

  <TextField
    {...f}
    lens={lastNameLens}
    label="Last name" />

  <TextField
    {...f}
    lens={addressStreetLens}
    label="Street" />

  <TextField
    {...f}
    lens={addressNumberLens}
    label="House number" />

  <pre>{JSON.stringify(person, null, 2)}</pre>
</>

With compose(), you can build type-safe forms for deeply-nested data structures.

The implementation of compose() looks complex, but it is worth looking at:

let compose = <T, S, F>(
  ts: Lens<T, S>,
  sf: Lens<S, F>
): Lens<T, F> => ({
  get: t => sf.get(ts.get(t)),
  set: (t, f) => ts.set(t, sf.set(ts.get(t), f))
});

Pay attention to the type signature: given a Lens<T, S> and a Lens<S, F>, return a Lens<T, F>. Once the type signature is in place, the implementation follows: the type system guides the implementation, and the type system will virtually guarantee correctness.

Handling lists

We already saw how to create a lens for a property of an object, using forProp():

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes"
};

firstNameLens.get(sherlock);
  // -> "Sherlock"

firstNameLens.set(sherlock, "Locky");
  // -> {
  //      firstName: "Locky",
  //      lastName: "Holmes"
  //    }

While handling properties of an object is done with forProp(), handling elements of a list can done with forIndex():

let hobbies = ["programming", "arguing"];

let first = forIndex<string>(0);

first.get(hobbies);
  // -> "programming"

first.set(hobbies, "coding");
  // -> ["coding", "arguing"]

In practice, though, forIndex() is not nearly as useful as forProp(). The forProp() function works well because we know the properties of an object ahead of time. For lists, on the other hand, the size is not known ahead of time, as lists can grow and shrink during execution.

To get a better understanding of how lists and lenses interact, let’s imagine a Person type with a list of hobbies:

type Person = {
  firstName: string;
  lastName: string;
  hobbies: string[];
};

We can create a lens for the list of hobbies:

let hobbiesLens: Lens<Person, string[]> =
  forProp<Person>()("hobbies");

A lens that focuses on a string[] is not directly useful, though. We’ll want to create one text field for each hobby, and a TextField component needs a lens that focuses on a string, not on a string[].

To access individual list elements, we need lenses for each list element. Rather than a single lens that focuses on a list of hobbies, we need a collection of lenses, each focusing on a single hobby:

let hobbyLenses: Lens<Person, string>[] =
  ??? // Implemented further down

Note the distinction in the type signature: hobbiesLens is one lens, while hobbyLenses is an array of lenses. While hobbiesLens focuses on a string array, hobbyLenses each focus on a single string.

To transform hobbiesLens into hobbyLenses, we need to know the number of elements in the list, so that we can generate the appropriate number of lenses. This is where makeArray() is useful:

let hobbyLenses: Lens<Person, string>[] =
  makeArray(hobbiesLens, person.hobbies.length);

We’ve left the implementation of makeArray() out, but it has this signature, in case you want to give implementing it a shot yourself (or check out the code in the demo):

(
  lens: Lens<T, F[]>,
  length: number
): Lens<T, F>[]

Once we have our list of lenses, we can create a TextField for each of these lenses:

let hobbiesLens = forProp<Person>()("hobbies");
let PersonForm = () => {
  let [person, setPerson] = useState(newPerson);
  let f = { top: person, setTop: setPerson };

  let hobbyLenses =
    makeArray(hobbiesLens, person.hobbies.length);

  return (
    <>
      {hobbyLenses.map(hobbyLens =>
        <TextField
          {...f}
          lens={hobbyLens}
          label="Hobby"
        />
      )}

      <pre>{JSON.stringify(person, null, 2)}</pre>
    </>
  );
};

We now have a form where we can edit existing list elements, but not add or remove any yet.

Adding and removing list elements

While the approach above works for modifying existing elements in a list, we need the ability to add new elements to a list and remove elements from a list.

To add an element to a list, we can use push(), whose type signature is (top: T, lens: Lens<T, F[]>, elem: F) => T:

let hobbiesLens =
  forProp<Person>()("hobbies");

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes",
  hobbies: ["deducing"]
}

push(sherlock, hobbiesLens, "sleuthing")
  // -> {
  //      firstName: "Sherlock",
  //      lastName: "Holmes",
  //      hobbies: ["deducing", "sleuthing"]
  //    }

In a React form, you could use push() as follows:

<button
  onClick={() =>
    setPerson(push(sherlock, hobbiesLens, ""))
  }
>New hobby</button>

The implementation of push() relies on over(), which applies a transformation over the value that the lens focuses on:

let over = <T, F>(
  top: T,
  lens: Lens<T, F>,
  fn: (f: F) => F
): T =>
  lens.set(top, fn(lens.get(top)));

The over() function is sometimes called transform() or map(). I prefer the latter personally, because it really feels like mapping. Here’s an example which transform’s Sherlock’s name into UPPERCASE!!!, for no particular reason:

let firstNameLens =
  forProp<Person>()("firstName");

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes"
}

over(
  person,
  firstNameLens,
  (a) => a.toUpperCase()
);
// -> {
//      firstName: "SHERLOCK",
//      lastName: "Holmes"
//    }

Once we have over(), we can implement push():

let push = <T, F>(
  top: T,
  lens: Lens<T, F[]>,
  elem: F
): T =>
  over(top, lens, elems => [...elems, elem]);

Now that we have push(), we can add new elements to a list. We are still lacking the ability to remove elements from a list, though. For this, there’s removeAt(), which removes an element at a specified index:

let hobbiesLens =
  forProp<Person>()("hobbies");

let sherlock: Person = {
  firstName: "Sherlock",
  lastName: "Holmes",
  hobbies: [
    "deducing",
    "sleuthing",
    "investigating",
  ]
}

removeAt(sherlock, hobbiesLens, 1)
  // -> {
  //      firstName: "Sherlock",
  //      lastName: "Holmes",
  //      hobbies: ["deducing", "investigating"]
  //    }

“Sleuthing” is a dated word. Good riddance, I say.

In a React form, you’d use removeAt() like this:

let hobbyLenses =
  makeArray(hobbiesLens, person.hobbies.length);

<>
  {hobbyLenses.map((hobbyLens, index) =>
    <>
      <TextField
        {...f}
        lens={hobbyLens}
        label="Hobby"
      />

      <button
        onClick={() =>
          setPerson(
            removeAt(sherlock, hobbiesLens, index)
          )
        }
      >Remove hobby</button>
    </>
  )}
</>

The removeAt() function can also implemented using over(), with some appropriate use of slice() to retain only the elements that are not to be removed:

let removeAt =
  <T, F>(
    top: T,
    lens: Lens<T, F[]>,
    idx: number
  ): T =>
    over(
      top,
      lens,
      elems => [
        ...elems.slice(0, idx),
        ...elems.slice(idx + 1)
      ]
    );

With all this in place, we have a type-safe way of managing lists.

Improving on single-select dropdowns

HTML provides the <select> element to create a dropdown list. Each element of this dropdown list, an <option>, has a value which identifies the selected option:

<select>
  <option></option>
  <option value="mari">Marina</option>
  <option value="star">Stardust</option>
  <option value="ruby">Ruby</option>
  <option value="sapp">Sapphire</option>
  <option value="elec">Electric</option>
  <option value="mint">Mint</option>
  <option value="slat">Slate</option>
</select>

If we wanted to give a Person a favorite color, we could add a string property:

type Person = {
  firstName: string;
  lastName: string;
  favoriteColor: string | null;
};

We could then use lenses to create a SingleSelectField, similar to our TextField. An implementation for this approach wouldn’t be too challenging.

There’s a limitation with this approach: the single-select dropdown values have to be strings. While this is common when building HTML single-select fields, we can do better, and treat dropdown values as full objects instead.

Imagine a Color type, and a Person whose favoriteColor property is a reference to a Color type:

type Color = {
  id: string;
  name: string;
  hex: string;
};

type Person = {
  firstName: string;
  lastName: string;
  favoriteColor: Color | null;
};

It’s beneficial to have a reference to Color rather than a string, because it allows us to encode additional information besides a value and a display label. For example, if a person has selected their favorite color, we can show a welcome message in their selected favorite color:

<div style={{ color: person.favoriteColor?.hex || "#000" }}>
  Hello, {person.firstName}!
</div>

The options for our single-select dropdown will be objects, and so we’ll need to define that list of objects as an array:

let allColors: Array<Color> = [
  { id: "mari", hex: "#0089a8", name: "Marina" },
  { id: "star", hex: "#e74132", name: "Stardust" },
  { id: "ruby", hex: "#bc1a50", name: "Ruby" },
  { id: "sapp", hex: "#45439d", name: "Sapphire" },
  { id: "elec", hex: "#c2d62e", name: "Electric" },
  { id: "mint", hex: "#29bc75", name: "Mint" },
  { id: "slat", hex: "#546173", name: "Slate" },
];

I can’t decide whether my favorite color is Stardust (so warm and powerful!) or Mint (so fresh and relaxing!).

We’ll also need a lens for a person’s favorite color:

let favoriteColorLens: Lens<Person, Color> =
  forProp<Person>()("favoriteColor");

We’ll create a DropdownField React component later on, but for now, let’s look at how we’d use it:

<DropdownField
  {...f}
  lens={favoriteColorLens}
  values={allColors}
  renderValue={m => m.name}
/>

The DropdownField has the usual properties top and setTop (passed in using {...f}), as well as lens, but there are two additional properties: the values property is the list of all option objects, and the renderValue property is a function that returns the display label:

interface DropdownFieldProps<T, F> {
  lens: Lens<T, F | null>;

  top: T;
  setTop: (top: T) => void;

  values: Array<F>;
  renderValue: (f: F) => string;
}

We’ll create a DropdownField React component, which will wrap a <select> HTML element. We will require value objects to have an id (hence the F extends { id: string }):

let DropdownField = <T, F extends { id: string }>(
  props: DropdownFieldProps<T, F>
) => {

This required id will be used as the value property of an <option> later on.

We’ll use the lens to get the currently-selected option:

  let value: F | null = props.lens.get(props.top);

We’ll need a callback function, for use later on, that is triggered when the <select> option changes. We can get the id of the currently-selected option, but we’ll have to loop through all options to find the one that matches:

  let onChange =
    (ev: React.ChangeEvent<HTMLSelectElement>) => {
      let id = ev.target.value;

      let value =
        props.values.find(v => v.id === id)
        || null;

      props.setTop(
        props.lens.set(props.top, value)
      );
    };

To render the <select> element, we loop over all values and generate <option> elements for each of them, and we set up the value and onChange properties of the <select> element:

  return (
    <select
      value={value?.id}
      onChange={onChange}
    >
      <option value=""></option>
      {props.values.map(value => (
        <option value={value.id} key={value.id}>
          {props.renderValue(value)}
        </option>
      ))}
    </select>
  );

Note that the DropdownField implementation has some assumptions built-in. There is always the empty option, and the value type is nullable (F | null, rather than just F). Additionally, each option object must have an id property. These assumptions might not hold in all situations.

Dropdowns versus radio buttons

Let’s take a moment to think about UX. A <select> element is a kind of single-select form field. Another kind of single-select form field is a set of radio buttons (<input type="radio">). In HTML, a select dropdown and a set of radio buttons is implemented quite differently, even though they serve a near-identical purpose: selecting one item from a collection.

We could define a RadioButtonsField component, whose usage resembles DropdownField:

<RadioButtonsField
  {...f}
  lens={favoriteColorLens}
  values={allColors}
  renderValue={m => m.name}
/>

The type signature of a RadioButtonsField is nearly identical to that of a DropdownField. The only difference is the type of the renderValue prop: for DropdownField, renderValue has to return a string, while for DropdownField, it can also return a JSX.Element.

This approach makes it trivial to swap a DropdownField for a RadioButtonsField. In plain-old HTML or React, this would not be the case, because dropdowns and radio buttons have wildly different implementations. Ideally, components with a similar purpose have a (near-)identical type signatures. This way, we can delay UX decisions around which kind of single-select field to use (dropdown or radio buttons), as we’ll be able to change one for the other trivially.

Future work

While lenses are an effective way of building forms, there are three areas where more research and development is needed to make lenses stand out as an approach to building React forms:

Validation: Lenses can be used in combination with common validation techniques, such as HTML’s built-in validation functionality and schema-based validation. However, these techniques don’t integrate neatly with lenses, and further research is needed to come with an approach where validation feels like a first-class concern.
Form helpers: While lenses themselves are simple, using them effectively for forms requires implementations for all types of form fields, from simple text fields to different types of multi-select lists. The demo contains the minimal implementation of some types of form fields, which ideally would grow and be properly packaged as an open-source package.
Performance: The performance of this particular implementation of lenses, and implementations of the form fields, has not been a cause for concern so far. Still, it is likely that situations will arise where the performance of lenses is just not adequate. More work needs to be done to ensure that lenses are an appropriate solution, even for unusually large and complex forms.

Closing thoughts

To me, lenses have proven their worth, and will have a prominent place in my toolbox for building forms. No other approach to building forms gives me the same development speed or gives me as much confidence.

With lenses, I can be confident that the forms I build work, with only a minimum of testing. The real-time IDE feedback and autocompletion means I can work faster, without compromising quality.

The approach outlined in this article will work with JavaScript as well, but you’ll not get the type-safety benefits. ↩
I make typign mistakes all the time, and have trouble detecting them — perhaps I’m mildly dyslexic? ↩
The double function invocation is necessary! When creating a lens using forProp, we need to specify the top type. If forProp only had a single function invocation, we’d have to specify both the top type and the focus type, because TypeScript does not (yet) support partial type inference. We’re working around this limitation by having two function invocations. See TypeScript issue #26242 for details. ↩

Denis Defreyne

Avoiding bugs in Ruby code using the state pattern

The example: A messaging platform

A naïve implementation

A safer implementation

Closing thoughts

The complete guide to implementing equality in Ruby

Why implementing equality matters

Entities

Value objects

Basic equality (double equals)

Basic equality for value objects

Basic equality for entities

Basic equality with type coercion

Strict equality

Hash equality

The #eql? method

The #hash method

Putting it together

Case equality (triple equals)

Ordered comparison

Wrapping up

Further reading

Using a modal lexer for parsing sublanguages

Ruby’s string scanner

A basic lexer

Testing it out

A modal lexer

The main lexer mode

The string lexer mode

The string interpolation lexer mode

Testing it out

Hints for the parser

Wrapping up

References

Drive change by writing structured proposal documents

Motivation

Proposed solution

Ownership

Lifecycle

Example

Advantages

Disadvantages

Open questions and concerns

Budget for replacing past purchases

Limited-lifetime items with non-trivial cost

How to budget for replacement

Starting out

Monthly budgeting

Lifetime vs. warranty

Advantages

Disadvantages

Horizontally scrolling multi-column layout: a retrospective

What worked

What didn’t work

Controlling column widths

Getting scrolling to work

Padding

Controlling column breaks

Open questions and future work

Closing thoughts

Using Lenses to build complex React forms in a type-safe way

The trouble with forms

Tightening the feedback loop using type safety

Demo app

What is a lens?

Lenses, more formally

Conveniently creating lenses with forProp()

Lenses for forms

Minimal form example

Prettier form example

Handling nested data by composing lenses

Handling lists

Adding and removing list elements

Improving on single-select dropdowns

Dropdowns versus radio buttons

Future work

Closing thoughts