First step for smart port facilities: Maintain fenders with drone and AI combination

With the advent of the fourth industrial revolution, there is an increasing need around the globe for the maintenance of port facilities by utilizing drones. Moreover, it has become more essential to ensure proactive maintenance of port facilities to secure their sustainable safety and serviceability since the number of aging port facilities in Republic of Korea, which are to exceed 30 years of service life by 2030, is expected to increase by about 50%.

In particular, it is critical in terms of port operations to ensure the safe docking of ships for loading and unloading purposes. Fenders perform a critical role in these situations. Fenders are installed on the sea side of the superstructure of quay wall to prevent damage on vessel hull and structure caused by the force of the ship berthing and frictional force. However, since most fenders are inaccessible via land directly, inspectors should commonly approach by using floating boats and visually inspect the condition of the fenders. It is very dangerous, time-consuming, and difficult to obtain detailed damage information due to sea waves and other risks.

The Korea Institute of Civil Engineering and Building Technology (KICT) has announced a new inspection approach to automatically detect fenders incorporating an AI model and a vision sensor on the unmanned aerial vehicle. It especially utilized a deep learning network with the densely connected encoder–decoder format. It is one of the networks widely used for pixel-level object detection, inspired by the eccentric function of the human vision.

The AI algorithm, developed by Department of Structural Engineering Research of KICT, research team led by Dr. Min, Jiyoung, was named “densely connected receptive field pyramid (DRFP)” or “tiny version of DRFP (DRFPt).” It aimed to precisely and quickly extract fenders in the pixel-level from numerous UAV images.

In order to efficiently search a wide area at once and to reduce the computational complexity, the standard convolution and the dilated convolution were densely connected in a pyramid form. And a dataset of fenders was collected by using UAV on various port facilities. The detection performance of the proposed model was compared to the other deep learning models in literature.

The results showed that the proposed model reliably detected fenders in images taken from various angles, with IoU and F1 scores exceeding 88%, despite changes in the color or shape caused by the tide. Here, IoU (Intersection over Union) means the ratio of the overlap area to the combined area of estimation and ground truth. F1 score is a statistical measure of the accuracy of a test. 100% means perfect overlap and accuracy.

There are numerous risk factors in every nook and cranny of port facilities that pose potential threats to the inspectors. Therefore, many port authorities are actively attempting to adopt new remote inspection technologies such as UAVs (unmanned aerial vehicles) and USVs (unmanned surface vehicles), both to ensure the safety of the inspectors and to facilitate their detailed and quantitative inspections on structural members that are hard to access via land. These unmanned vehicles are typically equipped with vision sensors through which they continually record video footage or single photographs as they continue to maneuver around the structure.

Considering the massive scale of port structures that extend many kilometers, the original data size of video recordings at high resolutions is usually too large for regular computers to manage. For example, about 4,000 aerial photographs taking up 50GB of storage were collected in a 1.25km stretch of capping concrete and main caisson structure at Incheon Port in Republic of Korea, which were captured by a 4k camera with 50% overlapping carried on a drone. Thus, to ensure effective management of the massive aerial photograph data over time, it is important to quickly extract only the target objects that require maintenance from the photos or videos and to store and manage the necessary quantitative information on the condition of the target objects.

Main researcher Dr. Min, Jiyoung said, “We are planning to upgrade this model to the fender health inspection system. It will enable us to quantitatively detect damage such as missing sections or cracks from only UAV images. This UAV-AI combination technology will automatically evaluate the fender serviceability in the future, securing the safety of inspectors and reducing the time cost in the field.”

Provided by
National Research Council of Science and Technology