Crossformer

class flood_forecast.transformer_xl.cross_former.Crossformer(n_time_series: int, forecast_history: int, forecast_length: int, seg_len: int, win_size=4, factor=10, d_model=512, d_ff=1024, n_heads=8, e_layers=3, dropout=0.0, baseline=False, n_targs=None, device=device(type='cuda', index=0))

__init__(n_time_series: int, forecast_history: int, forecast_length: int, seg_len: int, win_size=4, factor=10, d_model=512, d_ff=1024, n_heads=8, e_layers=3, dropout=0.0, baseline=False, n_targs=None, device=device(type='cuda', index=0))

Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting. https://github.com/Thinklab-SJTU/Crossformer

Parameters:

• n_time_series (int) – The total number of time series

• forecast_history (int) – The length of the input sequence

• forecast_length (int) – The number of steps to forecast

• seg_len (int) – Parameter specific to Crossformer, forecast_history must be divisible by seg_len

• win_size (int, optional) – The window size for the segment merge mechanism, defaults to 4 (original paper used 2)

• factor (int, optional) – _description_, defaults to 10

• d_model (int, optional) – _description_, defaults to 512

• d_ff (int, optional) – _description_, defaults to 1024

• n_heads (int, optional) – The number of heads in the multi-head attention mechanism, defaults to 8

• e_layers (int, optional) – The number of encoder layers, defaults to 3

• dropout (float, optional) – The amount of dropout to use when training the model, defaults to 0.0

• baseline (bool, optional) – A boolean of whether to use mean of the past time series , defaults to False

• device (str, optional) – _description_, defaults to torch.device(“cuda:0”)

forward(x_seq: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.SegMerging(d_model, win_size, norm_layer=<class 'torch.nn.modules.normalization.LayerNorm'>)

Segment Merging Layer.

The adjacent `win_size’ segments in each dimension will be merged into one segment to get representation of a coarser scale we set win_size = 2 in our paper

__init__(d_model, win_size, norm_layer=<class 'torch.nn.modules.normalization.LayerNorm'>)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

x: B, ts_d, L, d_model

class flood_forecast.transformer_xl.cross_former.scale_block(win_size, d_model, n_heads, d_ff, depth, dropout, seg_num=10, factor=10)

We can use one segment merging layer followed by multiple TSA layers in each scale the parameter `depth’ determines the number of TSA layers used in each scale We set depth = 1 in the paper .

__init__(win_size, d_model, n_heads, d_ff, depth, dropout, seg_num=10, factor=10)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.Encoder(e_blocks, win_size, d_model, n_heads, d_ff, block_depth, dropout, in_seg_num=10, factor=10)

The Encoder of Crossformer model.

__init__(e_blocks, win_size, d_model, n_heads, d_ff, block_depth, dropout, in_seg_num=10, factor=10)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.DecoderLayer(seg_len, d_model, n_heads, d_ff=None, dropout=0.1, out_seg_num=10, factor=10)

The decoder layer of Crossformer, each layer will make a prediction at its scale.

__init__(seg_len, d_model, n_heads, d_ff=None, dropout=0.1, out_seg_num=10, factor=10)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x, cross: Tensor)

x: the output of last decoder layer cross: the output of the corresponding encoder layer

class flood_forecast.transformer_xl.cross_former.Decoder(seg_len, d_layers, d_model, n_heads, d_ff, dropout, router=False, out_seg_num=10, factor=10)

The decoder of Crossformer, making the final prediction by adding up predictions at each scale.

__init__(seg_len, d_layers, d_model, n_heads, d_ff, dropout, router=False, out_seg_num=10, factor=10)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x, cross)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.FullAttention(scale=None, attention_dropout=0.1)

The Attention operation.

__init__(scale=None, attention_dropout=0.1)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(queries, keys, values)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.AttentionLayer(d_model, n_heads, d_keys=None, d_values=None, mix=True, dropout=0.1)

The Multi-head Self-Attention (MSA) Layer.

__init__(d_model, n_heads, d_keys=None, d_values=None, mix=True, dropout=0.1)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(queries, keys, values)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.TwoStageAttentionLayer(seg_num, factor, d_model, n_heads, d_ff=None, dropout=0.1)

The Two Stage Attention (TSA) Layer input/output shape: [batch_size, Data_dim(D), Seg_num(L), d_model]

__init__(seg_num, factor, d_model, n_heads, d_ff=None, dropout=0.1)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class flood_forecast.transformer_xl.cross_former.DSW_embedding(seg_len, d_model)

__init__(seg_len, d_model)

_summary_

Parameters:

• seg_len (_type_) – _description_

• d_model (_type_) – _description_

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.