Tutorial: Setting up a DAM project from pre-existing csv files

Introduction

This tutorial is an example of how a user would go about creating and running a DAM project from the python API. In this case the user uses a collection of csv files that would be normally be used as an input of the DAM UI. These csv files are:

• characteristicpoints.csv: download from here.

• locations.csv: download from here.

• scenarios.csv: download from here.

• segments.csv: download from here.

• soilmaterials.csv: download from here.

• soilprofiles.csv: download from here.

• surfacelines.csv: download from here .

Creating Segment class from segments.csv

In this example the segments.csv is imported by using pandas. The user can follow the following example to get the result of a list of Segments which can be later used as input of the DAM model.

In the segments.csv it can be seen that the calculation_type is given by string value. For example “Stability” or “Piping”. The first step to read the csv successfully is for the user to create a function that will transform this string input to the correct enumeration class dampythoninterface.segment.SegmentFailureMechanismTypeClass. The following code snippet shows how

def get_failure_mechanism(soil_profile):
    import dampythoninterface as dpi

    output_dictionary = {
        "All": dpi.SegmentFailureMechanismTypeClass.All,
        "Stability": dpi.SegmentFailureMechanismTypeClass.Stability,
        "Piping": dpi.SegmentFailureMechanismTypeClass.Piping,
        "Liquefaction": dpi.SegmentFailureMechanismTypeClass.Liquefaction,
    }
    return output_dictionary[soil_profile]

Then the user can read the csv file using pandas.

import pandas as pd
# read csv file using pandas
segment_csv = pd.read_csv("segments.csv", delimiter=";")

There are several ways to extract the values from the csv file. In this case the user loops through unique id segments. The second loop of the code lets the user loop through all the different rows of the csv file that have the same segment_id. For each segment a new list of probabilities is created.

import dampythoninterface as dpi
# initialize segment list that will be used for the dam input
segments_list = []
# Loop through all unique segment names of the csv
for segment_id in segment_csv["segment_id"].unique():
    # loop through all different probabilities and create a list of the SoilGeometryProbabilityElement
    list_of_probability_elements = []
    for csv_row in segment_csv[segment_csv["segment_id"] == segment_id].index:

In the second loop, a dictionary of the values that are contained in a certain row of the csv is created.

dictionary_probability_element = {
    "SoilProfileName": segment_csv["soilprofile_id"][csv_row],
    "SoilProfileType": dpi.SoilProfileTypeClass.ProfileType1D,
    "Probability": segment_csv["probability"][csv_row],
    "SegmentFailureMechanismType": get_failure_mechanism(
        segment_csv["calculation_type"][csv_row]
    ),
}

This dictionary can be used to initialize class dampythoninterface.segment.SoilGeometryProbabilityElement, which can be directly appended to the list_of_probability_elements of the segment

list_of_probability_elements.append(
    dpi.SoilGeometryProbabilityElement(
        **dictionary_probability_element
    )
)

After running the second loop the dampythoninterface.segment.Segment class is initiazed and appended to the segments list.

# create segment and append it to the segment list
segments_list.append(
    Segment(
        Name=str(segment_id),
        SoilGeometryProbability=list_of_probability_elements,
    )
)

The full code snippet for reading the segment.csv can be found in this document.

Creating Surface Line class by reading surfacelines.csv and characteristicpoints.csv

In this section the user will read the surfacelines.csv and characteristicpoints.csv and combine them to create a list of Surface Lines to input in the dam input. In this case two different functions are created the read_surface_lines_csv and read_characteristic_points_csv.

The read_surface_lines_csv has as input the location of the surfacelines.csv. First the surfacelines.csv is read using the pandas modules.

import pandas as pd
# read csv file using pandas
surface_lines_csv = pd.read_csv(file_name, delimiter=";")

An empty dictionary is initialized that will be filled with the points contained in the surfacelines.csv.

surface_lines = {}

Then a loop is defined that loops through each different location contained in the csv. The row that is defined for each location is extracted. Finally the row is reshaped so that every point is separated.

# loop though all locations
for location_id in surface_lines_csv["LOCATIONID"]:
    # get row of one of the locations
    row_csv = surface_lines_csv[surface_lines_csv["LOCATIONID"] == location_id]
    row_csv = row_csv.values[0][1:]
    # reshape row so that each point is separated
    row_csv = row_csv.reshape((-1, 3))

Finally, for each point location a list is initialized with the intention of appending all points in that certain location. Note that point that with a NaN value are not included. Also these points are not given a certain characteristic point type and therefore the enum type NONE is used from the dampythoninterface.surface_line.PointTypeEnum.

# create list of points all values all values have a none type
surface_lines[location_id] = []
for point in row_csv:
    # check if point is defined or if it is nan
    if not (str(point[0]) == "nan") or not (str(point[-1]) == "nan"):
        surface_lines[location_id].append(
            dpi.Point(X=point[0], Z=point[-1], PointType=dpi.PointTypeEnum.NONE))

Finally, the created function read_surface_lines_csv, returns a dictionary of location ids. Each location id is assigned a certain point list.

return surface_lines

After defining the surface line read function the read_characteristic_points function is defined. The first step is to establish a relationship between the columns of the characteristicpoints.csv and the dampythoninterface.surface_line.PointTypeEnum class. To do that a simple function is created where a helper dictionary is used to retrieve the enum value from the dampythoninterface.surface_line.PointTypeEnum class. The code is appended below

def get_characteristic_point_type(point_type: str):
    output_dictionary = {
        "Maaiveld buitenwaarts": dpi.PointTypeEnum.SurfaceLevelOutside,
        "Teen dijk buitenwaarts": dpi.PointTypeEnum.DikeToeAtRiver,
        "Kruin buitenberm": dpi.PointTypeEnum.ShoulderTopOutside,
        "Insteek buitenberm": dpi.PointTypeEnum.ShoulderBaseOutside,
        "Kruin buitentalud": dpi.PointTypeEnum.DikeTopAtRiver,
        "referentielijn": dpi.PointTypeEnum.DikeLine,
        "Verkeersbelasting kant buitenwaarts": dpi.PointTypeEnum.TrafficLoadOutside,
        "Verkeersbelasting kant binnenwaarts": dpi.PointTypeEnum.TrafficLoadInside,
        "Kruin binnentalud": dpi.PointTypeEnum.DikeTopAtPolder,
        "Insteek binnenberm": dpi.PointTypeEnum.ShoulderBaseInside,
        "Kruin binnenberm": dpi.PointTypeEnum.ShoulderTopInside,
        "Teen dijk binnenwaarts": dpi.PointTypeEnum.DikeToeAtPolder,
        "Insteek sloot dijkzijde": dpi.PointTypeEnum.DitchDikeSide,
        "Slootbodem dijkzijde": dpi.PointTypeEnum.BottomDitchDikeSide,
        "Slootbodem polderzijde": dpi.PointTypeEnum.BottomDitchPolderSide,
        "Insteek sloot polderzijde": dpi.PointTypeEnum.DitchPolderSide,
        "Maaiveld binnenwaarts": dpi.PointTypeEnum.SurfaceLevelInside,
    }
    return output_dictionary.get(point_type, dpi.PointTypeEnum.NONE)

Then the read_characteristic_points is defined. First, the csv file is read using the pandas module.

import pandas as pd
# read csv file using pandas
characteristic_points_csv = pd.read_csv(file_name, delimiter=";")

Then the column titles are extracted from the csv files and duplicate values are removed.

column_groups = [
    char_type_point.split("_")[-1]
    for char_type_point in characteristic_points_csv.columns[1:]
]
# remove duplicates
column_groups = list(dict.fromkeys(column_groups))

An empty dictionary is initialized that will be filled with the points contained in the characteristicpoints.csv.

characteristic_points_results = {}

Then a loop is defined that loops through each different location contained in the csv. The row that is defined for each location is extracted.

# loop though all locations
for characteristic_points in characteristic_points_csv["LOCATIONID"]:
    row_csv = characteristic_points_csv[
        characteristic_points_csv["LOCATIONID"] == characteristic_points
    ]

Then all characteristic points can be extracted. A loop is defined that will loop through all types of characteristic points. For each characteristic point type all relevant columns are extracted. This is actually a list of the X, Y and Z coordinate of a location.

characteristic_points_results[characteristic_points] = []
for characteristic_point_type in column_groups:
    # get relevant columns
    relevant_columns = [
        char_type_point
        for char_type_point in characteristic_points_csv.columns
        if characteristic_point_type in char_type_point
    ]

These columns are extracted from the row of the csv file and appended as a point to the dictionary of characteristic points. Points that have -1 coordinates are not included in the in the list of characteristic points.

char_point = row_csv[relevant_columns]
if not (char_point.isin([-1]).sum().sum() == 3):
    characteristic_points_results[characteristic_points].append(
        Point(
            X=list(char_point["X_" + characteristic_point_type])[0],
            Z=list(char_point["Z_" + characteristic_point_type])[0],
            PointType=get_characteristic_point_type(characteristic_point_type),
        )
    )

Finally, the created function read_characteristic_points, returns a dictionary of location ids. Each location id is assigned a certain point list.

return surface_lines

The final step includes combining the points of the locations found in the two different csv files. Note that the points need to be inputted in an ascending order. This is done as it can be seen in the following code snippet

def get_X(d):
    return d.X

# merge the two dictionaries to create surface line
for key, value in surface_lines.items():
    surface_lines[key] += characteristic_points.get(key, [])
    # sort points based on X coordinate
    surface_lines[key].sort(key=get_X)
# all the points of the surface lines are merged so the surface lines list can be created
surface_lines_dam_input = []
for location, points in surface_lines.items():
    surface_lines_dam_input.append(dpi.SurfaceLine(Name=location, Points=points))

The full code snippet for reading the csv files can be found in this document.

Creating profiles class from soilprofiles.csv file

To read the soilprofiles.csv the user can create a function that will create a list of dampythoninterface.soilprofile1D.SoilProfile1D class.

The profile csv is read using the pandas module

import pandas as pd
# read csv file using pandas
profile_csv = pd.read_csv(file_name, delimiter=";")

Then a loop is created that goes through all unique profile ids. A list of all 1D layers for a certain profile is initialized.

list_of_1D_layers = []

Then a second loop is created that loops all csv rows with a certain profile id. A dampythoninterface.soilprofile1D.Layer1D class is initialized for each row. If the soil type is sand the the IsAquifer property is set to True otherwise is set to False, otherwise the Piping calculation cannot be performed. This is append to the list of the 1D layers.

for csv_row in profile_csv[profile_csv["soilprofile_id"] == profile_id].index:
        if "zand" in profile_csv["soil_name"][csv_row]:
            is_aquifer = True
        else:
            is_aquifer = False
        list_of_1D_layers.append(
            dpi.Layer1D(
                **{
                    "Name": profile_csv["soil_name"][csv_row],
                    "SoilName": profile_csv["soil_name"][csv_row],
                    "TopLevel": profile_csv["top_level"][csv_row],
                    "IsAquifer": is_aquifer,
                    "WaterpressureInterpolationModel": dpi.WaterpressureInterpolationModelType.Automatic,
                }
            )
        )

The created list can be inputted in the profile list. That will be later used as dam input. Therefore, dampythoninterface.soilprofile1D.SoilProfile1D class is initialized. Note that the input of BottomLevel is hardcoded by the user and not extracted from a csv file.

# create profile and append it to the profile list
profile_list.append(
    dpi.SoilProfile1D(
        Name=str(profile_id),
        BottomLevel=-30,
        Layers1D=list_of_1D_layers,
    )
)
return profile_list

The full code snippet for reading the csv files can be found in this document.

Creating Location class from locations.csv

To create the list of dampythoninterface.location.Location which can later be inputted in the DamInput a function read_locations_csv is created. Two csv files are given as input, the locations.csv and the scenarios.csv. Firstly, these two files are read and merged based on the location_id csv column.

import dampythoninterface as dpi
import pandas as pd

# read csv file using pandas
location_csv = pd.read_csv(location_csv_name, delimiter=";")
scenario_csv = pd.read_csv(scenario_csv_name, delimiter=";")
merged_csv = pd.merge(location_csv, scenario_csv, on="location_id")

Then each row is read from the csv file.

# initialize location list that will be used for the dam input
locations_list = []
# Loop through all unique profile names of the csv
for index, row in merged_csv.iterrows():

The dampythoninterface.location.DesignScenario class is initialized. In this case there is only one design scenario.

# initialize DesignScenario
design_scenario = dpi.DesignScenario(
    Id=str(index),
    PolderLevel=row["polderlevel"],
    HeadPl2=row["head_pl2"],
    HeadPl3=row["head_pl3"],
    PlLineOffsetBelowDikeTopAtRiver=row["PLLineOffsetBelowDikeTopAtRiver"],
    PlLineOffsetBelowDikeTopAtPolder=row["PLLineOffsetBelowDikeTopAtPolder"],
    PlLineOffsetBelowShoulderBaseInside=row[
        "PLLineOffsetBelowShoulderBaseInside"
    ],
    PlLineOffsetBelowDikeToeAtPolder=row["PLLIneOffsetBelowDikeToeAtPolder"],
    RiverLevel=row["water_height"],
    RiverLevelLow=row["water_height_low"],
    DikeTableHeight=row["dike_table_height"],
    RequiredSafetyFactorStabilityInnerSlope=row[
        "safety_factor_stability_inner_slope"
    ],
    RequiredSafetyFactorStabilityOuterSlope=row[
        "safety_factor_stability_outer_slope"
    ],
    UpliftCriterionPiping=row["uplift_criterion_piping"],
    UpliftCriterionStability=row["uplift_criterion_stability"],
    RequiredSafetyFactorPiping=row["safety_factor_piping"],
)

The dampythoninterface.location.DesignOptions class is initialized.

# initialize design options class
design_options = dpi.DesignOptions(
    ShoulderEmbankmentMaterial=row["ophoogmateriaalberm"],
    StabilityShoulderGrowSlope=row["StabilityShoulderGrowSlope"],
    StabilityShoulderGrowDeltaX=row["StabilityShoulderGrowDeltaX"],
    StabilitySlopeAdaptionDeltaX=row["StabilitySlopeAdaptionDeltaX"],
    NewDikeTopWidth=row["NewDikeTopWidth"],
    NewDikeSlopeInside=row["NewDikeSlopeInside"],
    NewDikeSlopeOutside=row["NewDikeSlopeOutside"],
    NewShoulderTopSlope=row["NewShoulderTopSlope"],
    NewShoulderBaseSlope=row["NewShoulderBaseSlope"],
    NewMaxHeightShoulderAsFraction=row["NewMaxHeightShoulderAsFraction"],
    NewMinDistanceDikeToeStartDitch=row["NewMinDistanceDikeToeStartDitch"],
    UseNewDitchDefinition=row["UseNewDitchDefinition"],
    NewWidthDitchBottom=row["NewWidthDitchBottom"],
    NewDepthDitch=row["NewDepthDitch"],
    NewSlopeAngleDitch=row["NewSlopeAngleDitch"],
    RedesignDikeHeight=0,
    RedesignDikeShoulder=0,
    SlopeAdaptionStartCotangent=1,
    SlopeAdaptionEndCotangent=1,
    SlopeAdaptionStepCotangent=1,
    StabilityDesignMethod=dpi.StabilityDesignMethodType.OptimizedSlopeAndShoulderAdaption,
)

The dampythoninterface.location.StabilityOptions class is initialized.

# Initialize stability options
stability_options = dpi.StabilityOptions(
    MinimumCircleDepth=row["minimal_circle_depth"],
    TrafficLoad=row["trafficload"],
    ZoneType=get_zone_type(row["ZoneType"]),
)

The dampythoninterface.location.WaternetOptions class is initialized.

waternet_options = dpi.WaternetOptions(
    DampingFactorPl3=row["dempingsfactor_pl3"],
    DampingFactorPl4=row["dempingsfactor_pl4"],
    PhreaticLineCreationMethod=row["PLLineCreationMethod"],
    PenetrationLength=0,
    SlopeDampingFactor=1,
    IntrusionVerticalWaterPressure=dpi.IntrusionVerticalWaterPressureType.Standard,
    DikeSoilScenario=dpi.DikeSoilScenarioType.ClayDikeOnClay,
)

Finally, the dampythoninterface.location.Location class is initialized with all the inputs that were defined in the code snippets above.

# initialize location class
location = dpi.Location(
    XSoilGeometry2DOrigin=row["x_soilgeometry2D_origin"],
    SurfaceLineName=row["surfaceline_id"],
    SegmentName=row["segment_id"],
    DikeEmbankmentMaterial=row["ophoogmateriaaldijk"],
    Name=row["location_id"],
    DesignOptions=design_options,
    DesignScenarios=[design_scenario],
    StabilityOptions=stability_options,
    WaternetOptions=waternet_options,
)

The location is appended to the list of locations

locations_list.append(location)

The full code snippet for reading the csv files can be found in this document.

Creating Soil class from soilmaterials.csv

To create a list of dampythoninterface.soil.Soil class list that can later be inputted in the dam input a read_soilmaterials_csv function is created. This function takes as input the location od the soilmaterials csv file which is exported from the DamUI. The csv id read using pandas. And values of each row are used as inputs of the soil class. The reader is referred directly to the code snippet referred below.

The full code snippet for reading the csvs can be found in this document.

Setting up the input class and executing the calculation

Finally the user cab set up the more general settings of the dampythoninterface.stability_parameters.StabilityParameters class. An example is shown in the following code snippet. Note that despite the grid determination being Automatic the user DamPythonInterface tool cannot determine the Uplift Van grid values. Thus, they should be manually inputted.

# create stability parameters
stability_parameter = dpi.StabilityParameters(
    SearchMethod=dpi.SearchMethodType.Calculationgrid,
    GridDetermination=dpi.GridDeterminationType.Automatic,
    BishopTangentLinesDefinition=dpi.BishopTangentLinesDefinitionType.OnBoundaryLines,
    UpliftVanGridLeftVerticalPointsCount=36,
    UpliftVanGridLeftVerticalPointsDistance=2,
    UpliftVanGridLeftHorizontalPointsCount=14,
    UpliftVanGridLeftHorizontalPointsDistance=2,
    UpliftVanGridRightVerticalPointsCount=3,
    UpliftVanGridRightVerticalPointsDistance=2,
    UpliftVanGridRightHorizontalPointsCount=6,
    UpliftVanGridRightHorizontalPointsDistance=2,
    UpliftVanTangentLinesDefinition=0,
    UpliftVanTangentLinesDistance=0.25,
)

The dampythoninterface.input.DamInput class can then be defined as follows.

# create dam input
dam_input = dpi.DamInput(
    SurfaceLines=surface_lines_dam_input,
    Soils=soils,
    SoilProfiles1D=profiles,
    Segments=segments_list,
    Locations=locations,
    FailureMechanismSystemType=dpi.FailureMechanismSystem.StabilityInside,
    StabilityModelType=dpi.StabilityType.UpliftVan,
    ProjectPath=Path(""),
    StabilityParameters=stability_parameter,
)

The input xml path is defined and the xml is exported.

xml_dam_input = Path( "InputTutorialFile.xml")
dam_input.ExportToXml(xml_file=xml_dam_input)

The user can define an output file location and name and execute a DAM calculation

xml_dam_output = Path( "OutputTutorialxml.xml")
result_code = dam_input.execute(
    xml_input_file=xml_dam_input, xml_output_file=xml_dam_output
)

After the calculation is run the user can access the output by parsing the xml in Python. The user can also use the output. For example, in this case the user outputs the safety factor.

import xml.etree.cElementTree as et

# read output xml
tree = et.parse(str(xml_dam_output))
root = tree.getroot()
stability_results = (
    root.find("Results")
    .find("CalculationResults")
    .find("DesignResults")
    .find("StabilityDesignResults")
)
safety_factor = stability_results.get("SafetyFactor")
print("DAM produced a safety factor of ", safety_factor)

The full code snippet this document.