Data Science pet peeves: GRU calculation inconsistencies

As a data scientist diving deep into recurrent neural networks, you’ve probably encountered Gated Recurrent Units (GRUs). They’re everywhere in sequence modeling, from natural language processing to time series analysis. But here’s something that might drive you slightly mad: there are two different ways to write the GRU update equation, and both appear in respected publications and implementations.

GRU schematic (2015 Olah)

GRU schematic (2020 Bibi et al.)

When studying GRU architectures, you might notice something peculiar. Some papers and implementations use one formula for the hidden state update, while others use a seemingly different one. Let’s look at both:

Formulation 1 (2014 Cho et al.):

Formulation 2 (Common in modern frameworks):

At first glance, these equations might seem to represent different operations. In the first formula, (1-z_t) is applied to the previous hidden state, while in the second, it’s applied to the candidate hidden state. Which one is correct?

Here’s the beautiful thing: both formulations are mathematically equivalent! Let’s prove this by expanding both equations:

Expanding Formulation 1:

Expanding Formulation 2:

After rearranging terms, you can see that both equations yield the same result. This is a perfect example of how mathematical expressions can look different but represent the same underlying operation, especially when represented in schematics or diagrams!

Why Do We Have Two Formulations?

The existence of these two formulations stems from different ways of thinking about the update gate:

In the first formulation, (1-z_t) determines how much of the previous state to forget, while z_t controls how much of the new candidate state to add.

In the second formulation, z_t directly represents how much of the previous state to keep, while (1-z_t) determines how much of the new candidate state to incorporate.

This dichotomy has several practical implications:

1. Different deep learning frameworks might use different formulations, which can cause confusion when implementing GRUs from scratch or when comparing implementations.
2. When reading papers or documentation, you need to be aware of which formulation is being used to avoid misunderstandings.
3. The existence of two equivalent formulations can initially confuse students and practitioners learning about GRUs, especially lacking a formal mathematical background.

Best Practices

To avoid confusion (and make me happy personally ;-) , here are some recommendations:

1. When implementing GRUs, clearly document which formulation you’re using.
2. Stick to one formulation throughout your codebase.
3. Make sure you understand that both formulations are equivalent to avoid unnecessary debugging when comparing implementations like i did.

This inconsistency in GRU formulations is a perfect example of how mathematical notation can sometimes lead to apparent contradictions that aren’t really contradictions at all. While it might be a pet peeve, understanding this nuance actually deepens our comprehension of how GRUs work.

The next time you encounter a different GRU formulation than what you’re used to, remember: they’re both correct! It’s just a matter of perspective and how we choose to express the same mathematical concept.

Note: All equations in this article use the Hadamard (element-wise) product