A Software System for Extracting and Identifying Data from Camera Input

Introduction

Today, with advances in science and technology, almost everything can be done on a computer automatically through data collection and processing. Data input tools such as keyboards or scanners are gradually being replaced by more convenient devices such as touch screens, cameras, etc. Moreover, identification technology is one of the most commonly applied technologies today. Identity may include audio or visual recognition. Object recognition has many types such as speech, writing, face, bar code, etc. [1-4].. In parallel with the development of technology, the Artificial Intelligence (AI) market is also growing, in addition to many new companies being established, many large technology companies such as Apple, Google are also developing. there is a significant investment in AI research [5,6]. It can be said that AI is one of the great opportunities, completely changing the face of Vietnam's economy. With AI, the camera can identify and track information about a car in the city in just a few minutes [7,8]. After completion, this research result will create a basis for both theory and practical examples for other students to refer to, research and apply AI technology in many fields. Create a useful tool for both road users and managers. The software is highly applicable and practical in today's life.

Build System Design

Research Facilities: Identification is the process of classifying objects represented by a certain model and assigning them to a class (assigning the object a name) according to rules and standard patterns. The accreditation process based on known learning models is known as teacher accreditation or supervised learning (supervised learning); the opposite case is called unsupervised learning (non supervised learning) (Figure 1).

The idea of ​​this method: It is the license plate usually contains a uniform color, such as white and has a relatively certain area. So, it is possible to use the area development method, or use the moving.

The idea of ​​this approach is: The license plate is surrounded by a border. Therefore, it is possible to use the edge detection method, then use the Hough transform to extract the vertical and horizontal lines that exist in the image. The intersection of these lines is the area containing the license plate. And finally proceed to identify the characters on each sub-region.

Figure 2: Solution Diagram

Describe the Steps to be Taken

The flowchart of the steps is described as follows (Figure 2):

Input Image: Is an image with 256 gray levels, binary to a binary image. The purpose of the image binary phase is to highlight the license plate area of ​​the vehicle. When we separate the border, the area of ​​​​the license plate will appear clearly. Then use edge detection to get vertical to horizontal edges of the image. As a result of this process, the resulting image is a binary image containing only vertical and horizontal edges. Perform the Hough transform for the boundary segments just obtained and determine the line segments passing through the set of boundary points of each boundary, the result will be horizontal and vertical line segments. The intersection of these lines will create a subregion I_c.

Binary: The original image used is a 256 gray level image. Using 256 grayscale images does not reduce the versatility of the application. In fact, 256 grayscale images are still widely used and many video recorders are also capable of converting color images into 256 grayscale images. However, if we leave the image with 256 gray levels, edge detection is not efficient, because the continuous change of gray levels makes the boundary determination not easy and finding the continuous regions of the edge is quite difficult and limit. Therefore, we perform the binary image conversion to perform the edge extraction faster. So we get a binary image I_c, the resulting binary image is still guaranteed to separate the area containing the license plate from the surrounding area. It also removes regions that are uniform and less variable.

Boundary Layer Separation: Since the license plate has a border, we need to highlight the border (Boundary). Boundaries can be thought of as vertical and horizontal edges. The purpose of this stage is to separate the vertical and horizontal edges to find the subregion containing the license plate by calculating the intersection of the vertical and horizontal edges. Here, the input image is a binary image, so it is suitable for the first derivative method. Use two Sobel matrices in two directions x (vertical) and y (horizontal) to separate the edges of the image.

Hough Transform: The Hough transform is a method used to determine an approximate line (ellipse) passing through a set of points:

where, (x, y) is a point: y = mx +c => c = mx - y

Instead of finding N points on the line, one looks at all the points, see which point has the most lines passing through.

After constructing Hough lines, we obtain two sets of lines: A set of vertical lines and a set of horizontal lines of binary images with vertical and horizontal edges. Next, we define the intersection of these lines to form subregions that are quadrilaterals capable of containing license plates. However, the number of subregions obtained is very large. Therefore, instead of calculating the intersection of the lines, we will perform the intersection of the line segments. The main line segments are the segments passing through the set of points defined through the Hough line. Identifying these segments is simple through the two endpoints of the segment. Simply put, we can consider the upper (or left) end to be the point in the set of points lying on the line Hough whose sum of coordinates in the x and y axes is the smallest. The lower (or right) endpoint is the point in the set of points lying on the Hough line whose sum of coordinates in terms of x and y is the largest. To ensure that the vertical and horizontal lines (from the vertical and horizontal edge images) can intersect as in reality, we need to extend the line segments in two directions 5 points each. So the line we use is about 10 points longer than the actual line. The results of the Hough transform and the line segment extraction are shown in the figure below. Calculating the intersection of line segments is quite simple. We just need to calculate the intersection of the Hough lines and check if the intersection is high on the extracted line segment or not. The intersection of the lines will be the subregions I_c capable of containing license plates.

The result of the above gives us a set of subregions I_c that are quadrilaterals. At this point, we can confirm that the above problem is implemented by the Hough transform method. However, the difference (also an improvement point) of the method presented in this essay is: the Hough transform method only stops at this step and then performs the recognition of characters in the characters. subregion immediately. The identification can have many different methods, but for each subregion there is still a fairly large amount. So in this reseach, I don't just stop at finding the sub-regions Ic, but also find the sub-regions with the probability of containing the license plate, excluding the areas where the existence of the license plate is very small. So yes, the set of subregions Ic is narrowed, making the space of the identification problem narrower. Therefore, this workaround becomes faster, more efficient than the Hough transform approach, which does not depend much on the input image space.

Concept of License Plate Recognition System     

License plate recognition system is a system capable of analyzing images, performing checks and identifying license plates on vehicles, through video, video and image recording devices, from which it is able to assist in tracking Ask for more advanced information such as identifying information such as vehicle owner, registration information, vehicle tracking at slow speed, etc.

There are many image sources for the application. And developed, the image is directly acquired from the camera. In my report, I only stopped at identifying license plates (identifying letters) from photos.

There are different ways to categorize license plate recognition applications. One of the simplest ways is to categorize license plate recognition applications by intended use. License plate recognition applications can be divided into the following two categories:

Type 1: limited field of view:

• Input: Photos taken directly from digital image recording devices. Photographs recorded are usually limited to the area with license plates
• Working Principle: The vehicle must be driven at a slow enough speed for the camera to capture an image of the license plate area
• Applications: These license plate recognition applications are often used at checkpoints, toll stations, automatic parking lots and gatekeepers

Type 2: Unlimited field of view:

• Input: The input image is obtained from automatic recording devices, regardless of the angle and surrounding objects, the image is not required to only capture the area containing the license plate, but can be a composite image such as contain additional objects such as people, trees, streets, ... as long as the license plate area is clear enough to be able to recognize the characters in that area
• Working Principle: Due to the feature of not limiting the viewing area, the input image can be obtained from a recording device (camara, camera...). And so, the first job is to scan in the image, to determine which area is the license plate. Then perform region separation and identification. Finally, depending on the intended use, the identification results are transmitted or stored to serve the needs of the end user
• Application: Because it does not depend on the acquired image, it can be used in many places such as at traffic regulation places, at sensitive locations of traffic such as junctions and intersections. Control and detect violations of traffic safety

Product Design and Construction

Block Diagram of the System (Figure 3):

• Image Acquisition: Photos taken from the camera are entered into the computer as a database. Image is an analog signal
• Boundary Layer Separation: This block has the function of separating license plates from images taken by image processing methods
• Adjust Image Size: Adjust the image size when cropping number plates, so that it is suitable for segmentation and character recognition
• Character Segment: This block performs the function of separating each character in the number plate for character recognition. The image of each character is a black and white image
• Character Recognition: After segmenting the characters in the number plate and forming a character string. This string is included in the identifier block to perform character recognition in the string
• Recognition Results: After recognizing the characters in the license plate, we will output the recognition results through the recognition result block

Algorithm Flowchart

Software Introduction: Automatic license plate recognition software allows detecting and recognizing 2 types of license plates in Vietnam, rectangular and square plates from images, videos and webcams. The position of the camera is fixed and only one vehicle can drive through the gate at a time.

Figure 3: Block Diagram of the System

Figure 4: Algorithm Flowchart

Figure 5: Vehicle Border Identification Results at ICTU

Therefore, the system can only detect 1 plate per frame. Using different deep learning models and computer vision approaches. Basic motion capture has been performed to image moving vehicles using OpenCV where the threshold of the camera is fixed (threshold value varies according to the boundary area of the frame). If the vehicle hits the boundary, the image is captured and displayed on the screen. Recognition accuracy rate is 90%, performance: Common laptop, 3fps (cpu), 10fps (Intel Gpu) (Figure 4).

Datasets

With machine learning, users can improve the accuracy of the system by improving the training process. Use machine learning for the character recognition aspect, i.e. map a character image to its actual character and distinguish… In the model training directory there are several .txt, .pb files, which are used to train the character recognition model. In the project folder there is Python code used to train CNN and Plate Detection classes to recognize smart license plates. Creating accurate machine learning models capable of localizing and identifying multiple objects in a single image remains a core challenge in computer vision. The TensorFlow Object Detection API is an open source framework built with TensorFlow that makes it easy to build, train and deploy object detection models. Wrappers for languages like C, Python, Ruby and Java (using JavaCV) have been developed to encourage adoption by a wider audience. Recent releases have interfaces for C++.

Demonstration Project

The project is developed using Jupyter Notebook, Python 3.6 and 64 bit version, Ubuntu 64 bit version. The software is implemented using tensorflow, so the programmer must be familiar with Tensorflow and basic object detection and must also know the basic math to understand the tracking algorithm. The software is run on Ubuntu 18.04 so users must also be familiar with the Linux operating system and since all the code is done on Ubuntu.

Follow the steps:

• Open Terminal in the Application Menu of Ubuntu OS
• Type the command to change directory into the directory where the project is saved
• Create virtual environment python -m venv venv
• Activate virtual environment ./venv/bin/activate
• Install required pip packages
• Run the script: >> python DetectionPlate.py to start up

Result after Running the Program

The above program is installed in a simple form to create a program that can work with video streams that are specifically recorded here from the camera. It is much simpler in both functionality and interface. Users can self-develop more functions, many modules to create their own really interesting program based on the most basic things (Figure 5).