Designing a ‘neural puppeteer’ to recognize skeletal nodes

Credit: Computer Vision – ACCV 2022 (2023). DOI: 10.1007/978-3-031-26316-3_15

Imagine for a moment, that we are on a safari watching a giraffe graze. After looking away for a second, we then see the animal lower its head and sit down. But, we wonder, what happened in the meantime? Computer scientists from the University of Konstanz’s Centre for the Advanced Study of Collective Behaviour have found a way to encode an animal’s pose and appearance in order to show the intermediate motions that are statistically likely to have taken place.

One key problem in computer vision is that images are incredibly complex. A giraffe can take on an extremely wide range of poses. On a safari, it is usually no problem to miss part of a motion sequence, but, for the study of collective behavior, this information can be critical. This is where computer scientists with the new model “neural puppeteer” come in.

Predictive silhouettes based on 3D points

“One idea in computer vision is to describe the very complex space of images by encoding only as few parameters as possible,” explains Bastian Goldlücke, professor of computer vision at the University of Konstanz. One representation frequently used until now is the skeleton.

In a new paper published in the Proceedings of the 16th Asian Conference on Computer Vision, Bastian Goldlücke and doctoral researchers Urs Waldmann and Simon Giebenhain present a neural network model that makes it possible to represent motion sequences and render full appearance of animals from any viewpoint based on just a few key points. The 3D view is more malleable and precise than the existing skeleton models.

“The idea was to be able to predict 3D key points and also to be able to track them independently of texture,” says doctoral researcher Urs Waldmann. “This is why we built an AI system that predicts silhouette images from any camera perspective based on 3D key points.”






Credit: University of Konstanz

By reversing the process, it is also possible to determine skeletal points from silhouette images. On the basis of the key points, the AI system is able to calculate the intermediate steps that are statistically likely. Using the individual silhouette can be important. This is because, if you only work with skeletal points, you would not otherwise know whether the animal you’re looking at is a fairly massive one, or one that is close to starvation.

In the field of biology in particular, there are applications for this model: “At the Cluster of Excellence ‘Centre for the Advanced Study of Collective Behaviour’, we see that many different species of animals are tracked and that poses also need to be predicted in this context”, Waldmann says.

Long-term goal: Apply the system to as much data as possible on wild animals

The team started by predicting silhouette motions of humans, pigeons, giraffes and cows. Humans are often used as test cases in computer science, Waldmann notes. His colleagues from the Cluster of Excellence work with pigeons. However, their fine claws pose a real challenge. There was good model data for cows, while the giraffe’s extremely long neck was a challenge that Waldmann was eager to take on. The team generated silhouettes based on a few key points—from 19 to 33 in all.

Now the computer scientists are ready for the real world application: In the University of Konstanz’s Imaging Hanger, its largest laboratory for the study of collective behavior, data will be collected on insects and birds in the future. In the Imaging Hangar, it is easier to control environmental aspects such as lighting or background than in the wild. However, the long-term goal is to train the model for as many species of wild animals as possible, in order to gain new insight into the behavior of animals.






Credit: University of Konstanz

More information:
Simon Giebenhain et al, Neural Puppeteer: Keypoint-Based Neural Rendering of Dynamic Shapes, Computer Vision—ACCV 2022 (2023). DOI: 10.1007/978-3-031-26316-3_15

Provided by
University of Konstanz


Citation:
Designing a ‘neural puppeteer’ to recognize skeletal nodes (2023, March 8)
retrieved 8 March 2023
from https://techxplore.com/news/2023-03-neural-puppeteer-skeletal-nodes.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.

For all the latest Technology News Click Here 

 For the latest news and updates, follow us on Google News

Read original article here

Denial of responsibility! TechNewsBoy.com is an automatic aggregator around the global media. All the content are available free on Internet. We have just arranged it in one platform for educational purpose only. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials on our website, please contact us by email – [email protected]. The content will be deleted within 24 hours.