Model Evaluation

flood_forecast.evaluator.stream_baseline(river_flow_df: pandas.core.frame.DataFrame, forecast_column: str, hours_forecast=336) Tuple[pandas.core.frame.DataFrame, float][source]

Function to compute the baseline MSE by using the mean value from the train data.

flood_forecast.evaluator.plot_r2(river_flow_preds: pandas.core.frame.DataFrame) float[source]

We assume at this point river_flow_preds already has a predicted_baseline and a predicted_model column

flood_forecast.evaluator.get_model_r2_score(river_flow_df: pandas.core.frame.DataFrame, model_evaluate_function: Callable, forecast_column: str, hours_forecast=336)[source]

model_evaluate_function should call any necessary preprocessing

flood_forecast.evaluator.get_r2_value(model_mse, baseline_mse)[source]
flood_forecast.evaluator.get_value(the_path: str) None[source]
flood_forecast.evaluator.evaluate_model(model: Type[flood_forecast.time_model.TimeSeriesModel], model_type: str, target_col: List[str], evaluation_metrics: List, inference_params: Dict, eval_log: Dict) Tuple[Dict, pandas.core.frame.DataFrame, int, pandas.core.frame.DataFrame][source]

A function to evaluate a model. Called automatically at end of training. Can be imported for continuing to evaluate a model in other places as well.

from flood_forecast.evaluator import evaluate_model
forecast_model = PyTorchForecast(config_file)
e_log, df_train_test, f_idx, df_preds = evaluate_model(forecast_model, "PyTorch", ["cfs"], ["MSE", "MAPE"], {})
print(e_log) # {"MSE":0.2, "MAPE":0.1}
print(df_train_test) # will print a pandas dataframe
...

‘’’

flood_forecast.evaluator.infer_on_torch_model(model, test_csv_path: Optional[str] = None, datetime_start: datetime.datetime = datetime.datetime(2018, 9, 22, 0, 0), hours_to_forecast: int = 336, decoder_params=None, dataset_params: Dict = {}, num_prediction_samples: Optional[int] = None, probabilistic: bool = False, criterion_params: Optional[Dict] = None) Tuple[pandas.core.frame.DataFrame, torch.Tensor, int, int, flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, List[pandas.core.frame.DataFrame]][source]

Function to handle both test evaluation and inference on a test data-frame. :param model: The time series model present :param test_csv_path: The path to the test data-frame :return:

df: df including training and test data end_tensor: the final tensor after the model has finished predictions history_length: num rows to use in training forecast_start_idx: row index to start forecasting test_data: CSVTestLoader instance df_prediction_samples: has same index as df, and num cols equal to num_prediction_samples or no columns if num_prediction_samples is None

Return type

tuple()

flood_forecast.evaluator.handle_ci_multi(prediction_samples: torch.Tensor, csv_test_loader: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, multi_params: int, df_pred, decoder_param: bool, history_length: int, num_samples: int) List[pandas.core.frame.DataFrame][source]

Handles the CI confidence interval

Parameters
  • prediction_samples (torch.Tensor) – The number of predictions to generate

  • csv_test_loader (CSVTestLoader) – The test loader genreated in the previous

  • multi_params (int) – [description]

  • df_pred ([type]) – [description]

  • decoder_param (bool) – [description]

  • history_length (int) – [description]

  • num_samples (int) – [description]

Raises
  • ValueError – [description]

  • ValueError – [description]

Returns

Returns an array with different CI predictions

Return type

List[pd.DataFrame]

flood_forecast.evaluator.generate_predictions(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, targs=False, multi_params: int = 1) torch.Tensor[source]

A function to generate the actual model prediction

Parameters
  • model (Type[TimeSeriesModel]) – A PyTorchForecast

  • df (pd.DataFrame) – The main dataframe containing data

  • test_data (CSVTestLoader) – The test data loader

  • history (torch.Tensor) – The forecast historical data

  • device (torch.device) – The device usually cpu or cuda

  • forecast_start_idx (int) – The index you want the forecast to begin

  • forecast_length (int) – The length of the forecast the model outputs per forward pass

  • hours_to_forecast (int) – The number of time_steps to forecast in future

  • decoder_params (Dict) – The parameters the decoder function takes.

  • multi_params (int, optional) – n_targets, defaults to 1

Returns

The forecasted values for the time-series in a tensor

Return type

torch.Tensor

flood_forecast.evaluator.generate_predictions_non_decoded(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history_dim: torch.Tensor, forecast_length: int, hours_to_forecast: int) torch.Tensor[source]

Generates predictions for the models that do not use a decoder

Parameters
  • model (Type[TimeSeriesModel]) – A PyTorchForecast

  • df (pd.DataFrame) – [description]

  • test_data (CSVTestLoader) – [description]

  • history_dim (torch.Tensor) – [description]

  • forecast_length (int) – [description]

  • hours_to_forecast (int) – [description]

Returns

[description]

Return type

torch.Tensor

flood_forecast.evaluator.generate_decoded_predictions(model: Type[flood_forecast.time_model.TimeSeriesModel], test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, forecast_start_idx: int, device: torch.device, history_dim: torch.Tensor, hours_to_forecast: int, decoder_params: Dict, multi_targets: int = 1, targs: Union[bool, torch.Tensor] = False) torch.Tensor[source]
flood_forecast.evaluator.generate_prediction_samples(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, num_prediction_samples: int, multi_params=1, targs=False) numpy.ndarray[source]

ss