\chapter{Data format}\label{sec:USDataFormat}

Validation is done during three phases: import, editing in UI and for calculation.

\section{Validation during import}

All required and optional import data is placed in table OverviewDataUIAndEngine.xlsx, tab DAM\_input.

\ProgramName can import data from csv files and shape files
The sequence of import is
 1. csv files (except scenarios.csv), overwritten by:
	2. shape files, overwritten by:
		3. scenario.csv
If data is not present during import the default value is used, when a default is available. Default values are mentioned in OverviewDataUIAndEngine.xlsx

Required data
Data can be required when importing and/or required when calculating.
In OverviewDataUIAndEngine.xlsx is described what data is required and when. Example: crosssection is only required when using shapefiles (see column remark).\newline
If data is required for import and is missing or invalid, the exception handling is dependent of the kind of data, see \autoref{table:ExceptionHandlingReqData}.
Messages are given in the log window.

\begin{table}[H]
\begin{tabular}{|p{40mm}|p{\textwidth-40mm-24pt}|} \hline
\textbf{Parameter} & \textbf{Exception handling} \\ \hline
location$\_$id in location.csv (or via shape) & All locations with missing location$\_$id are not imported and message is given. \\ \hline
location$\_$id in scenarios.csv (or via shape) & All scenarios with missing location$\_$id are not imported and message is given. \\ \hline
surfaceline in location.csv$\_$id & All locations with missing surfaceline$\_$id are not imported and message is given. \\ \hline
Profielnaam in surfacelines.csv$\_$id & All surfacelines with missing surfaceline$\_$id are not imported and message is given. \\ \hline
segment$\_$id in locations.csv or via shape & All locations with missing segment$\_$id are not imported and message is given. \\ \hline
segment$\_$id in segment.csv & All locations using segments with missing segment$\_$id are not imported and message is given. \\ \hline
calculation$\_$type in segments.csv & All locations using segments with missing calculation$\_$type are not imported and message is given. \\ \hline
soilprofile$\_$id in segments.csv & All locations using segments with missing soilprofile$\_$id are not imported and message is given.  \\ \hline
top$\_$level in soilprofiles.csv& All locations linked (via segment) to soilprofiles with missing top$\_$level are not imported and message is given. \\ \hline
location$\_$scenario$\_$id & All scenarios with missing location$\_$scenario$\_$id are not imported and message is given. \\ \hline
characteristic points (required) & All locations with missing required characteristic points are not imported and message is given. \\ \hline
crosssection (via shape) & All locations with missing cross section are not imported and message is given. \\ \hline
SigmaTau curve (via csv) & When needed (as defined in the soils.csv) the curves must each meet the validation requirements as given in OverviewDataUIAndEngine.xlsx. All incorrect curves are not imported and a message is given. \\ \hline
\end{tabular}


\caption{Exception handling required data during import}
\label{table:ExceptionHandlingReqData}
\end{table}

In OverviewDataUIAndEngine.xls, tab DAM\_input also the optional data is listed. If these parameters are not present, no error or warning is given. When a default is available, this value is used, otherwise the parameters is NULL.
If the parameter is required for calculation and the value is NULL and the kernel does not provide a default, an error message is given.

\subsection{Editing characteristic points}\label{sec:ValCharPoints}
When the characteristic points are edited by the user in the UI, they must be validated by rules in following table. 	Blue text validations are not implemented yet. 

\begin{figure}[H]
	\centering
		\includegraphics[width=1\textwidth]{pictures/Figure_CharacteristicPointsInMacroStability.png}
	\caption{Characteristic points on the surface line}
	\label{fig:CharPoints}
\end{figure}


\begin{tabular}{|p{35mm}|p{35mm}|p{\textwidth-72pt-105mm}|p{13mm}|p{11mm}|p{11mm}|} \hline
\textbf{Name EN} & \textbf{Name NL} & \textbf{Symbol} & \textbf{Unit} & \textbf{Min value} & \textbf{Max value} \\ \hline
\multirow{2}{35mm}{Surface level outside} & \multirow{2}{35mm}{Maaiveld buitenwaarts} & X$_{C}$ & m & - & - \\ \cline{3-6}
 &  & Z$_{C}$ & m NAP & - & - \\ \hline
\multirow{2}{35mm}{Dike toe at river} & \multirow{2}{35mm}{Teen Dijk buitenwaarts} & X$_{D}$ & m & $>$X$_{C}$ & $<$X$_{E}$ \\ \cline{3-6}
 &  & Z$_{D}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{C}$} & $<$Z$_{G}$ \\ \hline
\multirow{2}{35mm}{Shoulder top outside} & \multirow{2}{35mm}{Kruin buitenberm} & X$_{E}$ & m & $>$X$_{D}$ & $<$X$_{F}$ \\ \cline{3-6}
 &  & Z$_{E}$ & m NAP &\textcolor[rgb]{0,0,1}{ $>$Z$_{D}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{F}$} \\ \hline
\multirow{2}{35mm}{Shoulder base outside} & \multirow{2}{35mm}{Insteek buitenberm} & X$_{F}$ & m & $>$X$_{E}$ & $<$X$_{G}$ \\ \cline{3-6}
 &  & Z$_{F}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{E}$ }& \textcolor[rgb]{0,0,1}{$<$Z$_{G}$ }\\ \hline
\multirow{2}{35mm}{Dike top at river} & \multirow{2}{35mm}{Kruin buitentalud} & X$_{G}$ & m & $>$X$_{F}$ & $<$X$_{H}$ \\ \cline{3-6}
 &  & Z$_{G}$ & m NAP & $>$Z$_{D}$ & - \\ \hline
\multirow{2}{35mm}{Traffic load outside} & \multirow{2}{35mm}{Verkeersbelasting buitenwaarts} & X$_{H}$ & m & \textcolor[rgb]{0,0,1}{$>$X$_{C}$ }& \textcolor[rgb]{0,0,1}{$<$X$_{I}$} \\ \cline{3-6}
 &  & Z$_{H}$ & m NAP &\textcolor[rgb]{0,0,1}{$>$Z$_{G}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{I}$} \\ \hline
\multirow{2}{35mm}{Traffic load inside} & \multirow{2}{35mm}{Verkeersbelasting binnenwaarts} & X$_{I}$ & m & \textcolor[rgb]{0,0,1}{$>$X$_{H}$ }& \textcolor[rgb]{0,0,1}{$<$X$_{S}$} \\ \cline{3-6}
 &  & Z$_{I}$ & m NAP &\textcolor[rgb]{0,0,1}{$>$Z$_{H}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{J}$} \\ \hline
\multirow{2}{35mm}{Dike top at polder} & \multirow{2}{35mm}{Kruin binnentalud} & X$_{J}$ & m & $>$X$_{I}$ & $<$X$_{K}$ \\ \cline{3-6}
 &  & Z$_{J}$ & m NAP & $>$Z$_{M}$ & -\\ \hline
\multirow{2}{35mm}{Shoulder base inside} & \multirow{2}{35mm}{Insteek binnenberm} & X$_{K}$ & m & $>$X$_{J}$ & $<$X$_{L}$ \\ \cline{3-6}
 &  & Z$_{K}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{J}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{L}$} \\ \hline
\multirow{2}{35mm}{Shoulder top inside} & \multirow{2}{35mm}{Kruin binnenberm} & X$_{L}$ & m & $>$X$_{K}$ & $<$X$_{M}$ \\ \cline{3-6}
 &  & Z$_{L}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{K}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{M}$ }\\ \hline
\multirow{2}{35mm}{Dike toe at polder} & \multirow{2}{35mm}{Teen dijk binnenwaarts} & X$_{M}$ & m & $>$X$_{L}$ & $<$X$_{N}$ \\ \cline{3-6}
 &  & Z$_{M}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{L}$} & $<$Z$_{J}$ \\ \hline
\multirow{2}{35mm}{Ditch dike side} & \multirow{2}{35mm}{Insteek sloot dijkzijde} & X$_{N}$ & m & $>$X$_{M}$ & $<$X$_{O}$ \\ \cline{3-6}
 &  & Z$_{N}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{M}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{O}$} \\ \hline
\multirow{2}{35mm}{Bottom ditch dike side} & \multirow{2}{35mm}{Slootbodem dijkzijde} & X$_{O}$ & m & $>$X$_{N}$ & $<$X$_{P}$ \\ \cline{3-6}
 &  & Z$_{O}$ & m NAP & $>$Z$_{N}$ & $<$Z$_{P}$ \\ \hline
\multirow{2}{35mm}{Bottom ditch polder side} & \multirow{2}{35mm}{Slootbodem polderzijde} & X$_{P}$ & m & $>$X$_{O}$ & $<$X$_{Q}$ \\ \cline{3-6}
 &  & Z$_{P}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{O}$} &\textcolor[rgb]{0,0,1}{ $<$Z$_{Q}$} \\ \hline
\multirow{2}{35mm}{Ditch polder side} & \multirow{2}{35mm}{Insteek sloot polderzijde} & X$_{Q}$ & m & $>$X$_{P}$ & $<$X$_{R}$ \\ \cline{3-6}
 &  & Z$_{Q}$ & m NAP & \textcolor[rgb]{0,0,1}{$>$Z$_{P}$} & \textcolor[rgb]{0,0,1}{$<$Z$_{R}$ }\\ \hline
\multirow{2}{35mm}{Surface level inside} & \multirow{2}{35mm}{Maaiveld binnenwaarts} & X$_{R}$ & m & - & - \\ \cline{3-6}
 &  & Z$_{R}$ & m NAP & - & - \\ \hline
\end{tabular}

If a required characteristic point is missing, (validation)message must be given: Characteristic point <Name> is missing.\newline
Required characteristic points are: 
\begin{itemize}
	\item Surface level outside
	\item	Dike toe at river
	\item Dike top at river
	\item Dike top at polder
	\item	Dike toe at polder
	\item Surface level inside
\end{itemize}

If a set of characteristic points is not complete, (validation)message must be given: Set Characteristic points <Name> is not complete. NL: Set karakteristieke <Naam> is niet compleet. \newline
Sets of characteristic points (Name and points from table) are:
\begin{itemize}
	\item TrafficLoad (H and I in table) NL: Verkeersbelasting
	\item ShoulderOutside (E and F in table) NL: Buitenberm
	\item ShoulderInside (K and L in table) NL: Binnenberm
	\item Ditch (N,O,P and Q in table) NL: Sloot
\end{itemize}

Note: All sets are optional, but when one of the characteristic points is asigned, the others of the set must also be present.

\section{Validation during editing}
\ProgramName validates during the import of the data and during editing by user in the User interface. Minimal and maximum values are given in OverviewDataUIAndEngine.xlsx.
The data is also validated by the used kernels when calculations are started.

\section{Validation for calculation}
\ProgramName validates during the import of the data and during editing by user in the User interface.
The data is also validated by the used kernels when calculations are started. Messages are given in the log window.
