Co-UDlabs WEBINAR :
Routine data validation in urban drainage – Principles and UDMT Toolbox applications
Friday 10 January 2025
2pm – 4pm (CEWT)
The webinar is only in English
INSA Lyon Deep and the Graie are partners of Co-UDlabs.
The webinar will introduce and illustrate the necessity of data validation in urban drainage and hydrology, will present some data validation rules and algorithms, and will show how to apply them with the UDMT (Urban Drainage Metrology Toolbox) free software developed in the Co-UDlabs H2020 European Project.
UMDT’s objectives ?
The UDMT aims to facilitate the application of best practices and methods in monitoring urban drainage systems. As these practices and methods are generic, they can also be applied in other domains.
The UDMT includes five blocks of functions:
– Sensor calibration / Correlation: this block provides various methods to determine i) calibration functions (based on a data set of outputs of a sensor submitted to standards or certified values), and ii) correlation functions (based on a data set of values given by a sensor and corresponding values obtained e.g. with laboratory analyses of samples).
– Calibration / Correlation correction: this block allows to convert raw values provided by a sensor into corrected values according to previously determined calibration or correlation functions. In addition, uncertainties in corrected values are estimated.
– Uncertainty assessment: this block allows to apply standard methods (type A, type B, Monte Carlo) to various data sets. In addition, the variograph method is proposed to estimate uncertainties in integrated values (e.g. sums, means, etc.).
– Data validation: this block provides a set of automated tests to help the user to validate data according to various criteria.
– Tracing experiments: this block allows to calculate a discharge from experimental data collected during tracing experiments. Tracing experiments are useful to qualify flowmeters in urban drainage systems.
About Co-UDlabs
Co-UDLABS (Building Collaborative Urban Drainage research lab communities) is a Horizon 2020 project funded under the Research Infrastructures programme (INFRAIA-02-2020 – Integrating Activities for Starting Communities).
Bringing together 9 partners offering access to 17 unique research facilities, Co-UDlabs offers training and free access to a wide range of high-level scientific instruments, smart monitoring technologies and digital water analysis tools for advancing knowledge and innovation in Urban drainage systems.
Report
Francois Clemens-Meyer and Jean-Luc Bertrand-Krajewski presented a two-hour webinar on data validation in urban hydrology on 10 January 2025. They illustrated the diversity of sources of error and uncertainty, the importance of data validation and the need to automate the pre-validation and to require user manual validation only for final decisions on validation. They then demonstrated the possibilities offered by the free software tool UDMT, developed as part of the Co-UDlabs project.
Generally speaking, urban hydrology measurements are complex, extremely site- and installation-dependent, and subject to high levels of uncertainty, as was pointed out during the discussions with the participants.
There are many sources of error and uncertainty, often unpredictable and sometimes improbable. Field investigations are a fundamental requirement before applying any digital validation method: proper installation and maintenance of the sensors, knowledge of practices and events upstream of the measurement point (experiments, maintenance operations, repairs, new illegal connections with specific discharges, root systems growing in the network, etc.).
To obtain high quality, usable information, a large quantity of data is required (in terms of spatial and temporal distribution); hence the interest, and the need, to automate of the validation process.
The key points for establishing a qualified dataset are :
- Data traceability: keep all the source data, even if this highlights the low proportion of data considered as validated! This is the price to pay!
- Knowledge of areas of validity: theoretical, local and expert.
- Redundancy of measurements or estimates (between several sensors or between sensors and modelling).
The data validation methods proposed in the UDMT toolbox are based on the characterisation of the areas of validity (theoretical, site characteristics and experts) and the gradients between different redundant measurements or assessments.
The qualification of the data is very different depending on the method used, but the initial question is “what data do we need for what purpose?”. Is it to provide an alert on high values, in which case the qualification of low values is of little importance; or is it to calculate flows, in which case high uncertainties on low values make the calculation very uncertain. Hence the importance of retaining all the data and analysing the results of the different validation methods, depending on the objectives.
While the work presented and the UDMT tool encourage the standardisation of methods and indicators for data validation, a discussion on the standardisation of the data itself has been initiated: this would be highly desirable, both for the development of knowledge and research, and for the effective use of data in the operation of structures!
We invite you to test UDMT on your data!
and contact the designers and developers if you need to: UrbanDrainageMetrologyToolbox@gmail.com