Big Data challenges - a multidisciplinary team approach

Abstract

The advent of smart grids created a surge in the quantity of data available to the DSO. Data has become an asset as crucial for grid operation as the grid per se. However, if the first concern is to have data, the second one is to have good data.This paper aims at demonstrating how data validation rules were built to extract value from Big Data using in-house tools and expertise. It presents a true account on a specific project that involved extracting, crunching and analysing data from multiple sources for a large volume of data and still proceeding when there were no guidelines for dealing with unexpected results.