The intelligent surveillance system of a security robot detects people and their faces.
- human recognition by cloud-based face recognition;
- alarms in case of human intrusion into a restricted unmanned area;
- PTZ camera tracking the moving people;
- saving face photos and recording video footages of the moving people.
To ensure total human recognition, each security robot must be connected to a face recognition provider’s server.
Face Recognition on the street
The first type of robots detects people at short distances, i.e. in the streets and in parks. The onboard supercomputer enables simultaneous human recognition in images from the robot’s six cameras. The cameras are positioned to form a full circle and can recognize human faces in all directions, no matter where the main flow of people moves.
The robot is as tall as a person of average height. Its video cameras match the face level of walking people. Such positioning of cameras enables to get human images that are optimal for recognition. The onboard supercomputer takes front face photos. By transferring solely front face photos, a security robot minimizes traffic in wireless communication channels and lessens load on the cloud-based face recognition servers.
SMP Robotics is open to integration with the total human recognition systems.
Human Tracking and Face Detection
Along with six cameras of the 360-degree panoramic CCTV system, the second robot type is fitted with a PTZ camera. Under control of a separate onboard supercomputer, the PTZ camera enables all-round scanning of the territory next to the robot. If a human is detected, the camera tracks the human being for a preset time or until a front face photo can be taken.
A separate onboard supercomputer ensures human / human face recognition, tracking with the PTZ camera on each frame in the video stream. The camera’s high frame processing speed enables a full turn of the PTZ camera in less than two minutes. Running in this mode, the robot can recognize people in the range of up to 100 meters.
Both robot types perform human search in a cloud-based database. Such solution enables to use one server for a group of robots. Moreover, each robot is an active query initiator and makes an independent decision whether to give the alarm once it gets the response.
Based Intelligent Surveillance System
In the robot’s intended use, the appearance of a person in the field of view of the robot’s cameras is perceived as an alarm event that demands attention of an operator. Therefore, the purpose of smart video analysis is to detect people on the images from the video surveillance system cameras.
To accomplish this task, a software has been developed that uses deep learning in recurrent neural networks. Such neural networks provide robust capabilities sufficient for real-world use. Even in complex lightning conditions encountered when the robot patrols the area, a neural network that has been trained on a large number of examples can recognize humans among numerous objects surrounding the robot.
Deep learning neural networks offer limitless capabilities for detecting other objects, such as cars, with the use of reinforcement learning algorithms.
Neural Networks on Embedded Computers
The robot’s video surveillance system includes a dedicated T9 supercomputer based on a Jetson TX2 processor module. NVIDIA’s Jetson TX2 offers high performance sufficient for simultaneous processing of 6 high-definition video streams with fairly low power consumption. Power consumption is a critical feature for an autonomous mobile robot that has to rely on built-in rechargeable batteries. The less power the video surveillance system needs, the longer the robot can work without recharging. The power consumption of a supercomputer based on Jetson TX2 is only 15W, so it can be used for long-term autonomous surveillance on a mobile robot.
The processor’s architecture is designed for working with deep learning neural networks. This provides high performance necessary for human detection in a video stream.
The embedded supercomputer is a perfect solution for detecting not only humans, but also other objects of interest for security purposes. For example, construction supplies are often stolen using pickup trucks. A neural network capable of recognizing a car in threatening proximity to a guarded facility will be able to give an alert of a potentially dangerous situation. For this feature to work, it is enough to train a neural network to recognize a particular vehicle type, and then add this neural network to the software running on the robot’s smart video surveillance system’s on-board computer. This approach does not require a hardware upgrade, and with the development of new infrastructural software it allows to quickly expand the security robots’ available functions.
Human Detection in Video Surveillance System
The detection of humans on the images obtained from six 360-degree cameras is performed by a single embedded computer. The detection range depends on numerous factors: the size of a person in the frame, whether the person’s full height is viewable, the lightning, the background, or whether there are other objects partially covering the person. Nevertheless, in the most general case, the detection range on the robots’ current models is around 40-50 meters. The image from each camera is analyzed with the frequency of several frames per second, which is sufficient for robust operation of the system.
In order to increase the human detection distance, for example when patrolling along chain-link fences, a potential threat can be localized at farther distances using a PTZ camera. This camera can detect a human at the distance of about 200 m. The camera works in scanning mode. It performs a 360-degree surveillance, rotating smoothly around its vertical axis. The PTZ camera image is processed by an additional dedicated Jetson TX2 supercomputer, that analyzes the high-definition video stream at 25 frames per second. The scanning detection mode can be suspended, allowing the PTZ camera to follow a specific object.
|Processor module||NVIDIA Jetson TX2|
|Video||8 AHD channels, 720p; SMA|
|Audio|| 2 channels; Hirose HR10;|
RS232, CAN; Hirose HR10
|Interfaces||RS232, CAN; Hirose HR10|
USB 3.0, USB 2.0, HDMI 2.0
Ethernet 10/100/1000 Mbps
|Power input voltage||2 VDC (10 … 16 VDC)|
|Power consumption||7 W max|
Hardware development kit for intelligent video surveillance
T9 development kit
Development of intelligent video surveillance and video analytics systems
T9 embedded computer
PROMT development kit
Development of intelligent video analytics systems for the S5.2 PROMT robot
T9 embedded computer +
360-degree view 6 camera systems
PICARD development kit
Development of intelligent video analytics systems for the S5.2 PICARD robot
2 pcs T9 embedded computers +
360-degree view 6 camera systems +
PTZ IP camera
and Big Data in Security System 5.2 +
As robots patrol the guarded premises, they accumulate data about places where they most frequently encounter potential threats. This data is processed and analyzed over large periods of time. As a result, a threat intensity map is composed based on external factors, time of day, weather, and various other information that provides us with Big Data. Processing this data can help predicting threats and enhance robotic patrolling at potentially dangerous times. It is also possible to use AI features at the data processing stage to automatically focus the patrolling robots on potentially dangerous directions. Furthermore, by analyzing Big Data accumulated during the robots’ use, we can reassign the patrolling routes and change the robots’ schedule.