allennlp.modules.highway

A Highway layer that does a gated combination of a linear transformation and a non-linear transformation of its input.

class allennlp.modules.highway.Highway(input_dim: int, num_layers: int = 1, activation: typing.Callable[torch.FloatTensor, torch.FloatTensor] = <function relu>) → None[source]

Bases: torch.nn.modules.module.Module

A Highway layer does a gated combination of a linear transformation and a non-linear transformation of its input. \(y = g * x + (1 - g) * f(A(x))\), where \(A\) is a linear transformation, \(f\) is an element-wise non-linearity, and \(g\) is an element-wise gate, computed as \(sigmoid(B(x))\).

This module will apply a fixed number of highway layers to its input, returning the final result.

Parameters:

input_dim : int

The dimensionality of \(x\). We assume the input has shape (batch_size, input_dim).

num_layers : int, optional (default=``1``)

The number of highway layers to apply to the input.

activation : Callable[[torch.Tensor], torch.Tensor], optional (default=``torch.nn.functional.relu``)

The non-linearity to use in the highway layers.

forward(inputs: torch.FloatTensor) → torch.FloatTensor[source]

Defines the computation performed at every call.

Should be overriden by all subclasses.