PyTorch Loaders

class flood_forecast.preprocessing.pytorch_loaders.CSVDataLoader(file_path: str, forecast_history: int, forecast_length: int, target_col: List[T], relevant_cols: List[T], scaling=None, start_stamp: int = 0, end_stamp: int = None, interpolate_param=True)[source]

Bases: torch.utils.data.dataset.Dataset

A data loader that takes a CSV file and properly batches for use in training/eval a PyTorch model :param file_path: The path to the CSV file you wish to use. :param forecast_history: This is the length of the historical time series data you wish to

utilize for forecasting
Parameters:
  • forecast_length – The number of time steps to forecast ahead (for transformer this must equal history_length)
  • relevant_cols – Supply column names you wish to predict in the forecast (others will not be used)
  • target_col – The target column or columns you to predict. If you only have one still use a list [‘cfs’]
  • scaling – (highly reccomended) If provided should be a subclass of sklearn.base.BaseEstimator

and sklearn.base.TransformerMixin) i.e StandardScaler, MaxAbsScaler, MinMaxScaler, etc) Note without a scaler the loss is likely to explode and cause infinite loss which will corrupt weights :param start_stamp int: Optional if you want to only use part of a CSV for training, validation

or testing supply these
Parameters:int (end_stamp) – Optional if you want to only use part of a CSV for training, validation, or testing supply these
inverse_scale(result_data: Union[torch.Tensor, pandas.core.series.Series, numpy.ndarray]) → torch.Tensor[source]
class flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader(df_path: str, forecast_total: int, use_real_precip=True, use_real_temp=True, target_supplied=True, interpolate=False, **kwargs)[source]

Bases: flood_forecast.preprocessing.pytorch_loaders.CSVDataLoader

Parameters:df_path (str) –

A data loader for the test data.

inverse_scale(result_data: Union[torch.Tensor, pandas.core.series.Series, numpy.ndarray]) → torch.Tensor
get_from_start_date(forecast_start: int)[source]
convert_real_batches(the_col: str, rows_to_convert)[source]

A helper function to return properly divided precip and temp values to be stacked with forecasted cfs.

convert_history_batches(the_col: Union[str, List[str]], rows_to_convert: pandas.core.frame.DataFrame)[source]

A helper function to return dataframe in batches of size (history_len, num_features)

Args:
the_col (str): column names rows_to_convert (pd.Dataframe): rows in a dataframe to be converted into batches