Validation of Data for Use in Civil Infrastructure Big Data Applications

Abstract

Big data applications are becoming more popular across many different fields and civil engineering is no different. The benefits of big data lie in its potential to provide valuable insights into various large datasets. Big data applications can identify patterns and trends that were previously unknown, which can help them make informed decisions and develop effective strategies. In the case of civil engineering, this could be taking large datasets that have been produced in relation to pieces of infrastructure and using them to create more efficient management strategies. One of the issues with using big data is that if the input dataset is flawed then the output resulting from any big data analysis will be compromised. Therefore, data validation is used, which is the process of ensuring that data is accurate, complete, and consistent. The consequence of not undertaking data validation may be Inaccurate or inconsistent data which can lead to incorrect insights and decisions in big data applications. This paper explores the necessity of validating data before it is used in a big data application. It outlines some of the different methods used for validating data and provides an overview of the potential issues that may arise from data validation errors. A case study is then presented showing the process of validation on data collected from four bridges and provides recommendations for implementing data validation as part of a larger big data workflow. The results of the case study show that validation of data is an important step in the big data process both for confidence in the outputs and to make big data applications more useful and more common in the civil engineering field. The discussion and presented case study in this paper highlight the necessity of validating data. It has shown some of the potential issues that may arise from not undertaking data validation.