Model Evaluation¶
-
flood_forecast.evaluator.
stream_baseline
(river_flow_df: pandas.core.frame.DataFrame, forecast_column: str, hours_forecast=336) -> (<class 'pandas.core.frame.DataFrame'>, <class 'float'>)[source]¶ Function to compute the baseline MSE by using the mean value from the train data.
-
flood_forecast.evaluator.
plot_r2
(river_flow_preds: pandas.core.frame.DataFrame) → float[source]¶ We assume at this point river_flow_preds already has a predicted_baseline and a predicted_model column
-
flood_forecast.evaluator.
get_model_r2_score
(river_flow_df: pandas.core.frame.DataFrame, model_evaluate_function: Callable, forecast_column: str, hours_forecast=336)[source]¶ model_evaluate_function should call any necessary preprocessing
-
flood_forecast.evaluator.
evaluate_model
(model: Type[flood_forecast.time_model.TimeSeriesModel], model_type: str, target_col: List[str], evaluation_metrics: List, inference_params: Dict, eval_log: Dict) → Tuple[Dict, pandas.core.frame.DataFrame, int, pandas.core.frame.DataFrame][source]¶ A function to evaluate a model. Called automatically at end of training. Can be imported for continuing to evaluate a model in other places as well.
from flood_forecast.evaluator import evaluate_model forecast_model = PyTorchForecast(config_file) e_log, df_train_test, f_idx, df_preds = evaluate_model(forecast_model, "PyTorch", ["cfs"], ["MSE", "MAPE"], {}) print(e_log) # {"MSE":0.2, "MAPE":0.1} print(df_train_test) # ...
‘’’
-
flood_forecast.evaluator.
infer_on_torch_model
(model, test_csv_path: Optional[str] = None, datetime_start: datetime.datetime = datetime.datetime(2018, 9, 22, 0, 0), hours_to_forecast: int = 336, decoder_params=None, dataset_params: Dict = {}, num_prediction_samples: Optional[int] = None, probabilistic: bool = False, criterion_params: Optional[Dict] = None) -> (<class 'pandas.core.frame.DataFrame'>, <class 'torch.Tensor'>, <class 'int'>, <class 'int'>, <class 'flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader'>, typing.List[pandas.core.frame.DataFrame])[source]¶ Function to handle both test evaluation and inference on a test data-frame. :return:
df: df including training and test data end_tensor: the final tensor after the model has finished predictions history_length: num rows to use in training forecast_start_idx: row index to start forecasting test_data: CSVTestLoader instance df_prediction_samples: has same index as df, and num cols equal to num_prediction_samples or no columns if num_prediction_samples is None
- Return type
tuple()
-
flood_forecast.evaluator.
handle_ci_multi
(prediction_samples: torch.Tensor, csv_test_loader: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, multi_params: int, df_pred, decoder_param: bool, history_length: int, num_samples: int) → List[pandas.core.frame.DataFrame][source]¶ [summary]
- Parameters
prediction_samples (torch.Tensor) – [description]
csv_test_loader (CSVTestLoader) – [description]
multi_params (int) – [description]
df_pred ([type]) – [description]
decoder_param (bool) – [description]
history_length (int) – [description]
num_samples (int) – [description]
- Raises
ValueError – [description]
ValueError – [description]
- Returns
Returns an array with different CI predictions
- Return type
List[pd.DataFrame]
-
flood_forecast.evaluator.
generate_predictions
(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, targs=False, multi_params: int = 1) → torch.Tensor[source]¶ A function to generate the actual model prediction
- Parameters
model (Type[TimeSeriesModel]) – A PyTorchForecast
df (pd.DataFrame) – The main dataframe containing data
test_data (CSVTestLoader) – The test data loader
history (torch.Tensor) – The forecast historical data
device (torch.device) – The device usually cpu or cuda
forecast_start_idx (int) – The index you want the forecast to begin
forecast_length (int) – The length of the forecast the model outputs per time step
hours_to_forecast (int) – The number of time_steps to forecast in future
decoder_params (Dict) – The parameters the decoder function takes.
multi_params (int, optional) – n_targets, defaults to 1
- Returns
The forecasted tensor
- Return type
torch.Tensor
-
flood_forecast.evaluator.
generate_predictions_non_decoded
(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history_dim: torch.Tensor, forecast_length: int, hours_to_forecast: int) → torch.Tensor[source]¶ Generates predictions for the models that do not use a decoder
- Parameters
model (Type[TimeSeriesModel]) – A PyTorchForecast
df (pd.DataFrame) – [description]
test_data (CSVTestLoader) – [description]
history_dim (torch.Tensor) – [description]
forecast_length (int) – [description]
hours_to_forecast (int) – [description]
- Returns
[description]
- Return type
torch.Tensor
-
flood_forecast.evaluator.
generate_decoded_predictions
(model: Type[flood_forecast.time_model.TimeSeriesModel], test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, forecast_start_idx: int, device: torch.device, history_dim: torch.Tensor, hours_to_forecast: int, decoder_params: Dict, multi_targets: int = 1, targs: Union[bool, torch.Tensor] = False) → torch.Tensor[source]¶
-
flood_forecast.evaluator.
generate_prediction_samples
(model: Type[flood_forecast.time_model.TimeSeriesModel], df: pandas.core.frame.DataFrame, test_data: flood_forecast.preprocessing.pytorch_loaders.CSVTestLoader, history: torch.Tensor, device: torch.device, forecast_start_idx: int, forecast_length: int, hours_to_forecast: int, decoder_params: Dict, num_prediction_samples: int, multi_params=1, targs=False) → numpy.ndarray[source]¶ ss