Model Evaluation

flood_forecast.evaluator.stream_baseline(river_flow_df: pandas.core.frame.DataFrame, forecast_column: str, hours_forecast=336) -> (<class 'pandas.core.frame.DataFrame'>, <class 'float'>)

Function to compute the baseline MSE by using the mean value from the train data.

flood_forecast.evaluator.plot_r2(river_flow_preds: pandas.core.frame.DataFrame) → float

We assume at this point river_flow_preds already has a predicted_baseline and a predicted_model column

flood_forecast.evaluator.get_model_r2_score(river_flow_df: pandas.core.frame.DataFrame, model_evaluate_function: Callable, forecast_column: str, hours_forecast=336)

model_evaluate_function should call any necessary preprocessing

flood_forecast.evaluator.get_r2_value(model_mse, baseline_mse)

flood_forecast.evaluator.get_value(the_path: str) → None

flood_forecast.evaluator.evaluate_model(model: Type[flood_forecast.time_model.TimeSeriesModel], model_type: str, target_col: List[str], evaluation_metrics: List, inference_params: Dict, eval_log: Dict) → Tuple[Dict, pandas.core.frame.DataFrame, int, pandas.core.frame.DataFrame]

A function to evaluate a model. Called automatically at end of training. Can be imported for continuing to evaluate a model in other places as well.

from flood_forecast.evaluator import evaluate_model
forecast_model = PyTorchForecast(config_file)
e_log, df_train_test, f_idx, df_preds = evaluate_model(forecast_model, "PyTorch", ["cfs"], ["MSE", "MAPE"], {})
print(e_log) # {"MSE":0.2, "MAPE":0.1}
print(df_train_test) #
...

‘’’

flood_forecast.evaluator.infer_on_torch_model(model, test_csv_path: Optional[str] = None, datetime_start: datetime.datetime = datetime.datetime(2018, 9, 22, 0, 0), hours_to_forecast: int = 336, decoder_params=None, dataset_params: Dict = {}, num_prediction_samples: Optional[int] = None, probabilistic: bool = False, criterion_params: Optional[Dict] = None) -> (<class 'pandas.core.frame.DataFrame'>, <class 'torch.Tensor'>, <class 'int'>, <class 'int'>, <class 'flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader'>, typing.List[pandas.core.frame.DataFrame])

Function to handle both test evaluation and inference on a test data-frame. :return:

df: df including training and test data end_tensor: the final tensor after the model has finished predictions history_length: num rows to use in training forecast_start_idx: row index to start forecasting test_data: CSVTestLoader instance df_prediction_samples: has same index as df, and num cols equal to num_prediction_samples or no columns if num_prediction_samples is None

Return type

tuple()

flood_forecast.evaluator.handle_ci_multi(prediction_samples: torch.Tensor, csv_test_loader: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, multi_params: int, df_pred, decoder_param: bool, history_length: int, num_samples: int) → List[pandas.core.frame.DataFrame]

[summary]

Parameters

• prediction_samples (torch.Tensor) – [description]

• csv_test_loader (CSVTestLoader) – [description]

• multi_params (int) – [description]

• df_pred ([type]) – [description]

• decoder_param (bool) – [description]

• history_length (int) – [description]

• num_samples (int) – [description]

Raises

• ValueError – [description]

• ValueError – [description]

Returns

Returns an array with different CI predictions

Return type

List[pd.DataFrame]

flood_forecast.evaluator.generate_predictions(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, targs=False, multi_params: int = 1) → torch.Tensor

A function to generate the actual model prediction

Parameters

• model (Type[TimeSeriesModel]) – A PyTorchForecast

• df (pd.DataFrame) – The main dataframe containing data

• test_data (CSVTestLoader) – The test data loader

• history (torch.Tensor) – The forecast historical data

• device (torch.device) – The device usually cpu or cuda

• forecast_start_idx (int) – The index you want the forecast to begin

• forecast_length (int) – The length of the forecast the model outputs per time step

• hours_to_forecast (int) – The number of time_steps to forecast in future

• decoder_params (Dict) – The parameters the decoder function takes.

• multi_params (int, optional) – n_targets, defaults to 1

Returns

The forecasted tensor

Return type

torch.Tensor

flood_forecast.evaluator.generate_predictions_non_decoded(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history_dim: torch.Tensor, forecast_length: int, hours_to_forecast: int) → torch.Tensor

Generates predictions for the models that do not use a decoder

Parameters

• model (Type[TimeSeriesModel]) – A PyTorchForecast

• df (pd.DataFrame) – [description]

• test_data (CSVTestLoader) – [description]

• history_dim (torch.Tensor) – [description]

• forecast_length (int) – [description]

• hours_to_forecast (int) – [description]

Returns

[description]

Return type

torch.Tensor

flood_forecast.evaluator.generate_decoded_predictions(model: Type[flood_forecast.time_model.TimeSeriesModel], test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, forecast_start_idx: int, device: torch.device, history_dim: torch.Tensor, hours_to_forecast: int, decoder_params: Dict, multi_targets: int = 1, targs: Union[bool, torch.Tensor] = False) → torch.Tensor

flood_forecast.evaluator.generate_prediction_samples(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, num_prediction_samples: int, multi_params=1, targs=False) → numpy.ndarray

ss