Big data analytics aplicada en la integración de datos de internet de las cosas, caso de uso: Agricultura de precisión

Abstract

The need for greater agricultural production with excellent quality parameters is increasing, something that has been an additional problem for farmers, causing them to apply the concept of precision agriculture PA, which consists of providing greater support for crop management thanks to the analysis of data collected through technologies and electronic media. This is the reason for this project, which consists of analyzing large amounts of data pertaining to crop lots, applying data collection, transformation and integration processes. The entire architecture proposed for the development of the prototype is based on each of the phases steps of the MAMBO methodology, which was proposed by a team of strategic consultants specialized in the field of analytics and who saw the need to implement improvements in the CRISP-DM methodology for Big Data Analytics projects. It starts with a general analysis of precision agriculture systems and the data required to be analyzed in the future, then for the data acquisition process, and considering the lack of access to real historical data, a wireless sensor network simulator is used to generate synthetic data as messages of two types, configuration and readings, the first, for information related to crops, devices, companies, among others, and the second, for values generated by the sensors. An IoT middleware is also used to transfer messages through IoT protocols, MQTT and Apache Kafka, and store them in relational and non-relational databases. For data analysis it is not possible to use only the set of data generated by the sensors as these simply contain identifiers and values of each sensor, for this, it is necessary to relate them with the information of the configurations that are stored in the relational database, this process is called data integration, for this the Denodo tool is used, The problem is that the data that is integrated is not housed in a physical space and also, its volume increases considerably to the point of having a Big Data, that is why an Apache Hadoop cluster is necessary as a Big Data infrastructure for the storage and processing of large amounts of data through its HDFS and YARN services. The application of basic, descriptive and predictive statistics is performed to the dataset stored in the cluster by means of Python and R programming languages, a major focus is on prediction models in time series belonging to predictive statistics, and therefore, in the evaluation of the prototype, Two scenarios are established, the first one consists of measuring transfer times for loading streaming data, i.e., messages from the time they are generated by the simulator to their storage in HDFS, and on the other hand, the second scenario consists of evaluating the prediction models by applying quality metrics. The results of the first scenario indicate that, for large volumes of data, the integration times are high compared to the transfer times of individual messages, this, considering the resources of the machine where the entire architecture of the prototype is executed, while, for the second scenario, many of the prediction models evaluated, presented similar results to each other, this means that the models, for the most part, adjusted well to the real data set to make significant predictions for the decision making process within an agricultural process.