%% bare_jrnl.tex %% V1.4b %% 2015/08/26 %% by Michael Shell %% see http://www.michaelshell.org/ %% for current contact information. %% %% This is a skeleton file demonstrating the use of IEEEtran.cls %% (requires IEEEtran.cls version 1.8b or later) with an IEEE %% journal paper. %% %% Support sites: %% http://www.michaelshell.org/tex/ieeetran/ %% http://www.ctan.org/pkg/ieeetran %% and %% http://www.ieee.org/ %%************************************************************************* %% Legal Notice: %% This code is offered as-is without any warranty either expressed or %% implied; without even the implied warranty of MERCHANTABILITY or %% FITNESS FOR A PARTICULAR PURPOSE! %% User assumes all risk. %% In no event shall the IEEE or any contributor to this code be liable for %% any damages or losses, including, but not limited to, incidental, %% consequential, or any other damages, resulting from the use or misuse %% of any information contained here. %% %% All comments are the opinions of their respective authors and are not %% necessarily endorsed by the IEEE. %% %% This work is distributed under the LaTeX Project Public License (LPPL) %% ( http://www.latex-project.org/ ) version 1.3, and may be freely used, %% distributed and modified. A copy of the LPPL, version 1.3, is included %% in the base LaTeX documentation of all distributions of LaTeX released %% 2003/12/01 or later. %% Retain all contribution notices and credits. %% ** Modified files should be clearly indicated as such, including ** %% ** renaming them and changing author support contact information. ** %%************************************************************************* % *** Authors should verify (and, if needed, correct) their LaTeX system *** % *** with the testflow diagnostic prior to trusting their LaTeX platform *** % *** with production work. The IEEE's font choices and paper sizes can *** % *** trigger bugs that do not appear when using other class files. *** *** % The testflow support page is at: % http://www.michaelshell.org/tex/testflow/ \documentclass[journal]{IEEEtran} % % If IEEEtran.cls has not been installed into the LaTeX system files, % manually specify the path to it like: % \documentclass[journal]{../sty/IEEEtran} % Some very useful LaTeX packages include: % (uncomment the ones you want to load) % *** MISC UTILITY PACKAGES *** % %\usepackage{ifpdf} % Heiko Oberdiek's ifpdf.sty is very useful if you need conditional % compilation based on whether the output is pdf or dvi. % usage: % \ifpdf % % pdf code % \else % % dvi code % \fi % The latest version of ifpdf.sty can be obtained from: % http://www.ctan.org/pkg/ifpdf % Also, note that IEEEtran.cls V1.7 and later provides a builtin % \ifCLASSINFOpdf conditional that works the same way. % When switching from latex to pdflatex and vice-versa, the compiler may % have to be run twice to clear warning/error messages. % *** CITATION PACKAGES *** % %\usepackage{cite} % cite.sty was written by Donald Arseneau % V1.6 and later of IEEEtran pre-defines the format of the cite.sty package % \cite{} output to follow that of the IEEE. Loading the cite package will % result in citation numbers being automatically sorted and properly % "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using % cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's % \cite will automatically add leading space, if needed. Use cite.sty's % noadjust option (cite.sty V3.8 and later) if you want to turn this off % such as if a citation ever needs to be enclosed in parenthesis. % cite.sty is already installed on most LaTeX systems. Be sure and use % version 5.0 (2009-03-20) and later if using hyperref.sty. % The latest version can be obtained at: % http://www.ctan.org/pkg/cite % The documentation is contained in the cite.sty file itself. % *** GRAPHICS RELATED PACKAGES *** % \ifCLASSINFOpdf % \usepackage[pdftex]{graphicx} % declare the path(s) where your graphic files are % \graphicspath{{../pdf/}{../jpeg/}} % and their extensions so you won't have to specify these with % every instance of \includegraphics % \DeclareGraphicsExtensions{.pdf,.jpeg,.png} \else % or other class option (dvipsone, dvipdf, if not using dvips). graphicx % will default to the driver specified in the system graphics.cfg if no % driver is specified. % \usepackage[dvips]{graphicx} % declare the path(s) where your graphic files are % \graphicspath{{../eps/}} % and their extensions so you won't have to specify these with % every instance of \includegraphics % \DeclareGraphicsExtensions{.eps} \fi % graphicx was written by David Carlisle and Sebastian Rahtz. It is % required if you want graphics, photos, etc. graphicx.sty is already % installed on most LaTeX systems. The latest version and documentation % can be obtained at: % http://www.ctan.org/pkg/graphicx % Another good source of documentation is "Using Imported Graphics in % LaTeX2e" by Keith Reckdahl which can be found at: % http://www.ctan.org/pkg/epslatex % % latex, and pdflatex in dvi mode, support graphics in encapsulated % postscript (.eps) format. pdflatex in pdf mode supports graphics % in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure % that all non-photo figures use a vector format (.eps, .pdf, .mps) and % not a bitmapped formats (.jpeg, .png). The IEEE frowns on bitmapped formats % which can result in "jaggedy"/blurry rendering of lines and letters as % well as large increases in file sizes. % % You can find documentation about the pdfTeX application at: % http://www.tug.org/applications/pdftex % *** MATH PACKAGES *** % %\usepackage{amsmath} % A popular package from the American Mathematical Society that provides % many useful and powerful commands for dealing with mathematics. % % Note that the amsmath package sets \interdisplaylinepenalty to 10000 % thus preventing page breaks from occurring within multiline equations. Use: %\interdisplaylinepenalty=2500 % after loading amsmath to restore such page breaks as IEEEtran.cls normally % does. amsmath.sty is already installed on most LaTeX systems. The latest % version and documentation can be obtained at: % http://www.ctan.org/pkg/amsmath % *** SPECIALIZED LIST PACKAGES *** % %\usepackage{algorithmic} % algorithmic.sty was written by Peter Williams and Rogerio Brito. % This package provides an algorithmic environment fo describing algorithms. % You can use the algorithmic environment in-text or within a figure % environment to provide for a floating algorithm. Do NOT use the algorithm % floating environment provided by algorithm.sty (by the same authors) or % algorithm2e.sty (by Christophe Fiorio) as the IEEE does not use dedicated % algorithm float types and packages that provide these will not provide % correct IEEE style captions. The latest version and documentation of % algorithmic.sty can be obtained at: % http://www.ctan.org/pkg/algorithms % Also of interest may be the (relatively newer and more customizable) % algorithmicx.sty package by Szasz Janos: % http://www.ctan.org/pkg/algorithmicx % *** ALIGNMENT PACKAGES *** % %\usepackage{array} % Frank Mittelbach's and David Carlisle's array.sty patches and improves % the standard LaTeX2e array and tabular environments to provide better % appearance and additional user controls. As the default LaTeX2e table % generation code is lacking to the point of almost being broken with % respect to the quality of the end results, all users are strongly % advised to use an enhanced (at the very least that provided by array.sty) % set of table tools. array.sty is already installed on most systems. The % latest version and documentation can be obtained at: % http://www.ctan.org/pkg/array % IEEEtran contains the IEEEeqnarray family of commands that can be used to % generate multiline equations as well as matrices, tables, etc., of high % quality. % *** SUBFIGURE PACKAGES *** %\ifCLASSOPTIONcompsoc % \usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig} %\else % \usepackage[caption=false,font=footnotesize]{subfig} %\fi % subfig.sty, written by Steven Douglas Cochran, is the modern replacement % for subfigure.sty, the latter of which is no longer maintained and is % incompatible with some LaTeX packages including fixltx2e. However, % subfig.sty requires and automatically loads Axel Sommerfeldt's caption.sty % which will override IEEEtran.cls' handling of captions and this will result % in non-IEEE style figure/table captions. To prevent this problem, be sure % and invoke subfig.sty's "caption=false" package option (available since % subfig.sty version 1.3, 2005/06/28) as this is will preserve IEEEtran.cls % handling of captions. % Note that the Computer Society format requires a larger sans serif font % than the serif footnote size font used in traditional IEEE formatting % and thus the need to invoke different subfig.sty package options depending % on whether compsoc mode has been enabled. % % The latest version and documentation of subfig.sty can be obtained at: % http://www.ctan.org/pkg/subfig % *** FLOAT PACKAGES *** % %\usepackage{fixltx2e} % fixltx2e, the successor to the earlier fix2col.sty, was written by % Frank Mittelbach and David Carlisle. This package corrects a few problems % in the LaTeX2e kernel, the most notable of which is that in current % LaTeX2e releases, the ordering of single and double column floats is not % guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a % single column figure to be placed prior to an earlier double column % figure. % Be aware that LaTeX2e kernels dated 2015 and later have fixltx2e.sty's % corrections already built into the system in which case a warning will % be issued if an attempt is made to load fixltx2e.sty as it is no longer % needed. % The latest version and documentation can be found at: % http://www.ctan.org/pkg/fixltx2e %\usepackage{stfloats} % stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e % the ability to do double column floats at the bottom of the page as well % as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in % LaTeX2e). It also provides a command: %\fnbelowfloat % to enable the placement of footnotes below bottom floats (the standard % LaTeX2e kernel puts them above bottom floats). This is an invasive package % which rewrites many portions of the LaTeX2e float routines. It may not work % with other packages that modify the LaTeX2e float routines. The latest % version and documentation can be obtained at: % http://www.ctan.org/pkg/stfloats % Do not use the stfloats baselinefloat ability as the IEEE does not allow % \baselineskip to stretch. Authors submitting work to the IEEE should note % that the IEEE rarely uses double column equations and that authors should try % to avoid such use. Do not be tempted to use the cuted.sty or midfloat.sty % packages (also by Sigitas Tolusis) as the IEEE does not format its papers in % such ways. % Do not attempt to use stfloats with fixltx2e as they are incompatible. % Instead, use Morten Hogholm'a dblfloatfix which combines the features % of both fixltx2e and stfloats: % % \usepackage{dblfloatfix} % The latest version can be found at: % http://www.ctan.org/pkg/dblfloatfix %\ifCLASSOPTIONcaptionsoff % \usepackage[nomarkers]{endfloat} % \let\MYoriglatexcaption\caption % \renewcommand{\caption}[2][\relax]{\MYoriglatexcaption[#2]{#2}} %\fi % endfloat.sty was written by James Darrell McCauley, Jeff Goldberg and % Axel Sommerfeldt. This package may be useful when used in conjunction with % IEEEtran.cls' captionsoff option. Some IEEE journals/societies require that % submissions have lists of figures/tables at the end of the paper and that % figures/tables without any captions are placed on a page by themselves at % the end of the document. If needed, the draftcls IEEEtran class option or % \CLASSINPUTbaselinestretch interface can be used to increase the line % spacing as well. Be sure and use the nomarkers option of endfloat to % prevent endfloat from "marking" where the figures would have been placed % in the text. The two hack lines of code above are a slight modification of % that suggested by in the endfloat docs (section 8.4.1) to ensure that % the full captions always appear in the list of figures/tables - even if % the user used the short optional argument of \caption[]{}. % IEEE papers do not typically make use of \caption[]'s optional argument, % so this should not be an issue. A similar trick can be used to disable % captions of packages such as subfig.sty that lack options to turn off % the subcaptions: % For subfig.sty: % \let\MYorigsubfloat\subfloat % \renewcommand{\subfloat}[2][\relax]{\MYorigsubfloat[]{#2}} % However, the above trick will not work if both optional arguments of % the \subfloat command are used. Furthermore, there needs to be a % description of each subfigure *somewhere* and endfloat does not add % subfigure captions to its list of figures. Thus, the best approach is to % avoid the use of subfigure captions (many IEEE journals avoid them anyway) % and instead reference/explain all the subfigures within the main caption. % The latest version of endfloat.sty and its documentation can obtained at: % http://www.ctan.org/pkg/endfloat % % The IEEEtran \ifCLASSOPTIONcaptionsoff conditional can also be used % later in the document, say, to conditionally put the References on a % page by themselves. % *** PDF, URL AND HYPERLINK PACKAGES *** % %\usepackage{url} % url.sty was written by Donald Arseneau. It provides better support for % handling and breaking URLs. url.sty is already installed on most LaTeX % systems. The latest version and documentation can be obtained at: % http://www.ctan.org/pkg/url % Basically, \url{my_url_here}. % *** Do not adjust lengths that control margins, column widths, etc. *** % *** Do not use packages that alter fonts (such as pslatex). *** % There should be no need to do such things with IEEEtran.cls V1.6 and later. % (Unless specifically asked to do so by the journal or conference you plan % to submit to, of course. ) % correct bad hyphenation here \hyphenation{op-tical net-works semi-conduc-tor} \usepackage{graphicx} \graphicspath{ {./figures/} } \usepackage{multirow} \begin{document} % % paper title % Titles are generally capitalized except for words such as a, an, and, as, % at, but, by, for, in, nor, of, on, or, the, to and up, which are usually % not capitalized unless they are the first or last word of the title. % Linebreaks \\ can be used within to get better formatting as desired. % Do not put math or special symbols in the title. \title{REAL-TIME FACE DETECTION AND HUMAN TRACKING SYSTEM ON FPGA CYCLONE-V} % % % author names and IEEE memberships % note positions of commas and nonbreaking spaces ( ~ ) LaTeX will not break % a structure at a ~ so this keeps an author's name from being broken across % two lines. % use \thanks{} to gain access to the first footnote area % a separate \thanks must be used for each paragraph as LaTeX2e's \thanks % was not built to handle multiple paragraphs % %\author{Michael~Shell,~\IEEEmembership{Member,~IEEE,} % John~Doe,~\IEEEmembership{Fellow,~OSA,} % and~Jane~Doe,~\IEEEmembership{Life~Fellow,~IEEE}% <-this % stops a space %\thanks{J. Doe and J. Doe are with Anonymous University.}% <-this % stops a space %\thanks{Manuscript received April 19, 2005; revised August 26, 2015.}} \author{Huu Luong NGUYEN, Minh Son NGUYEN, and Tri Nhut DO*% <-this % stops a space \thanks{This work was supported in part by the University of Information Technology under Grant BS123456} \thanks{Huu Luong NGUYEN is with Vietnam National University Ho Chi Minh City, HCMC 700000 Vietnam. He is now with the Faculty of Computer Engineering, University of Information Technology, HCMC 700000 Vietnam e-mail: (luongnh@uit.edu.vn).}% <-this % stops a space \thanks{Minh Son NGUYEN is with Vietnam National University Ho Chi Minh City, HCMC 700000 Vietnam. He is now with the Faculty of Computer Engineering, University of Information Technology, HCMC 700000 Vietnam e-mail: (sonnm@uit.edu.vn).}% <-this % stops a space \thanks{Tri Nhut DO is with Vietnam National University Ho Chi Minh City, HCMC 700000 Vietnam. He is now with the Faculty of Computer Engineering, University of Information Technology, HCMC 700000 Vietnam e-mail: (*trinhutdo@gmail.com, trinhutdo@uit.edu.vn).}% <-this % stops a space \thanks{Manuscript received April 19, 2021; revised August 26, 2021.}} % note the % following the last \IEEEmembership and also \thanks - % these prevent an unwanted space from occurring between the last author name % and the end of the author line. i.e., if you had this: % % \author{....lastname \thanks{...} \thanks{...} } % ^------------^------------^----Do not want these spaces! % % a space would be appended to the last name and could cause every name on that % line to be shifted left slightly. This is one of those "LaTeX things". For % instance, "\textbf{A} \textbf{B}" will typeset as "A B" not "AB". To get % "AB" then you have to do: "\textbf{A}\textbf{B}" % \thanks is no different in this regard, so shield the last } of each \thanks % that ends a line with a % and do not let a space in before the next \thanks. % Spaces after \IEEEmembership other than the last one are OK (and needed) as % you are supposed to have spaces between the names. For what it is worth, % this is a minor point as most people would not even notice if the said evil % space somehow managed to creep in. % The paper headers \markboth{REV Journal on Electronics and Communications}% {Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for IEEE Journals} % The only time the second header will appear is for the odd numbered pages % after the title page when using the twoside option. % % *** Note that you probably will NOT want to include the author's *** % *** name in the headers of peer review papers. *** % You can use \ifCLASSOPTIONpeerreview for conditional compilation here if % you desire. % If you want to put a publisher's ID mark on the page you can do it like % this: %\IEEEpubid{0000--0000/00\$00.00~\copyright~2015 IEEE} % Remember, if you use this you must call \IEEEpubidadjcol in the second % column for its text to clear the IEEEpubid mark. % use for special paper notices %\IEEEspecialpapernotice{(Invited Paper)} % make the title area \maketitle % As a general rule, do not put math, special symbols or citations % in the abstract or keywords. \begin{abstract} Face detection in image sequence (real-time video stream) has been an active research area in the computer vision field in recent years due to its potential applications such as surveillance cameras, human computer interfaces, smart rooms, intelligent robots and biomedical image analysis. Face detection is a process that determines whether an image has a face or not. In this paper, an embedded system for detecting and tracking human faces in real-time video stream implemented on FPGA DE10-NANO is proposed. The system can be divided into two parts: data streaming, data processing. Experimental results show that the system is capable of accurately detecting faces of up to 5 different people at a distance of up to 1.5 meters from the camera, coexisting in the same frame in resolution of 320x240 pixels with a detection speed of only several hundred milliseconds prove the feasibility of the system. A comparison with similar existing projects will be discussed for evaluation and conclusion as well. \end{abstract} % Note that keywords are not normally used for peerreview papers. \begin{IEEEkeywords} %IEEE, IEEEtran, journal, \LaTeX, paper, template. Embedded system, human tracking system, face detection system, FPGA. \end{IEEEkeywords} % For peer review papers, you can put extra information on the cover % page as needed: % \ifCLASSOPTIONpeerreview % \begin{center} \bfseries EDICS Category: 3-BBND \end{center} % \fi % % For peerreview papers, this IEEEtran command inserts a page break and % creates the second title. It will be ignored for other modes. \IEEEpeerreviewmaketitle \section{Introduction} % The very first letter is a 2 line initial drop letter followed % by the rest of the first word in caps. % % form to use if the first word consists of a single letter: % \IEEEPARstart{A}{demo} file is .... % % form to use if you need the single drop letter followed by % normal text (unknown if ever used by the IEEE): % \IEEEPARstart{A}{}demo file is .... % % Some journals put the first two words in caps: % \IEEEPARstart{T}{his demo} file is .... % % Here we have the typical use of a "T" for an initial drop letter % and "HIS" in caps to complete the first word. %\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file'' %for IEEE journal papers produced under \LaTeX\ using %IEEEtran.cls version 1.8b and later. % You must have at least 2 lines in the paragraph with the drop letter % (should never be an issue) \IEEEPARstart{F}{ield} programmable gate arrays (FPGAs) [1] have become extremely popular in almost all the application domains such as computer vision, object detection and tracking. The FPGA is an ideal device to carry out works related to Video and Image processing by the parallel processing capability. Paul Viola and Michael Jones introduced in [2] an effective method for object detection using Haar feature-based cascade classifiers. Objects in images are detected based on a machine learning algorithm in which a cascade function is trained from a lot of positive and negative images. A Virtex-II 2V1000 using a MicroBlaze processor for face detection employed Viola-Jones algorithm introduced by Vinod Nair, Pierre-Olivier Laprise and James J. Clark in [3]. The results show that the system can detect people accurately at a rate of about 2.5 frames per second when it is running at 75MHz, communicating with dedicated hardware over FSL links. In addition, another system for face detection introduced by Hichem Ben Fakih, Ahmed Elhossini and Ben Juurlink in [4] was also employing Viola-Jones algorithm as well. The proposed design is able to discover faces in real-time with high accuracy. Speed-up is achieved by exploiting the parallelism in the design, where multiple classifier cores can be added. To maintain a flexible design, classifier cores can be assigned to different images. Moreover using different training data, every core is able to detect a different object type. The Zynq-7000 SoC from Xilinx is used, which features an ARM Cortex-A9 dual-core CPU and a programmable logic (FPGA). The current implementation focuses on the face detection and achieves a real-time detection at the rate of 16.53 FPS on image resolution of 640x480 pixels, which represents a speed-up of 6.46 times compared to the equivalent OpenCV software solution. That’s why the Viola-Jones algorithm is a well-known method for face detection systems utilizing Xilinx FPGA devices. Recently, the Haar feature-based cascade classifier was widely applied on processing to identify and count the oil palm trees [5]; to detect vehicles on moving for monitoring, avoiding accident and regulating traffic [6]; to recognize and verify handwritten signature with the accuracy of 92\% for many different types of writers' languages and style [7]; to detect the defects of five types on different kinds of textiles [8]. In this paper, Viola-Jones algorithm using Haar-like features for face detection is employed in the proposed system. All images captured from the OV7670 camera module and stored in Image Frame Buffer will be processed for face detection. The proposed system is implemented on DE10-Nano Development Board [9]. This board has a robust hardware design platform built around the Intel System-on-Chip (SoC) FPGA with 5,570 Kbits embedded memory, 110K programmable logic elements. Besides, it also supports other components such as: processor, peripherals, high-speed DDR3 memory, analog to digital capabilities, Ethernet networking, and much more that allow users to have not only simple but also flexible designs or even a high-performance, low-power processor system. The board top view along with its components descriptions are shown in Fig. \ref{fig: F1}. \begin{figure}[h] \centering \includegraphics[width=0.45\textwidth]{Fig1Top} \caption{DE10-Nano Development Board} \label{fig: F1} \end{figure} The rest of this paper is organized as follows. Designs of the proposed system followed by a block diagram for face detection based on Viola-Jones algorithm using Haar-like features are described in section II. Section III describes the hardware system and software setup, an evaluation of experimental results is presented as well. Section IV concludes the paper with future directions. \section{PROPOSED SYSTEM} % %\hfill mds % %\hfill August 26, 2015 \subsection{System Model} The design of the proposed system is made up of three main components: the OV7670 camera, the DE10-Nano development board, and a HDMI monitor (as shown in Fig. \ref{fig: F2}). The processing step includes frame capture, integral image calculation, parallel sub window processing. \begin{figure}[h] \centering \includegraphics[width=0.5\textwidth]{Fig2} \caption{Overview of the proposed system} \label{fig: F2} \end{figure} The OV7670 camera is connected to the DE10-Nano development board employing I2C protocol (see Fig. \ref{fig: F3}). The communication between them is via SDA and SCL pins where the DE10-Nano development board acts as Master and the OV7670 camera acts as Slave. Images captured by the OV7670 camera will be converted into RGB444 format (ADC 12bits). Captured images are stored in the captured Block and then will be copied into the face detection Block in order to apply the Viola- Jones algorithm. \begin{figure}[h] \centering \includegraphics[width=0.43\textwidth]{Fig3} \caption{I2C protocol for camera sensor and SoC Cyclone-V in DE10-Nano} \label{fig: F3} \end{figure} The FPGA consists of a number of components. These components are image frame capture, processing and display. All of these components are synchronized and controlled from a control unit. They are listed as in Table \ref{tab: T1}. A phase lock loop (PLL) generates 100Mhz clock signals that are contributed to each component for synchronous processing in high speed between modules in the system. Fig. \ref{fig: F4} shows The block diagram of the real-time detection and tracking algorithms on FPGA Cyclone – V of DE10 Nano Board. \begin{table}[h!] \centering \caption{FPGA RESOURCES} \begin{tabular}{l l} \hline\hline %$\begin{array}{ll} Family & Cyclone V \\ Device & 5CSEBA6U23I7 \\ Logic utilization (in ALMs) & 10,865/41,910 (26 \%) \\ Total registers & 16284 \\ Total pins & 60 / 314 (19 \%) \\ Total block memory bits & 2,826,805/5,662,720 (50\%) \\ Total DSP Blocks & 87 / 112 (78 \%) \\ Total PLLs & 1 / 6 (17 \%) \\ %\end{array}$ \hline\hline \end{tabular} \label{tab: T1} \end{table} \begin{figure}[h] \centering \includegraphics[width=0.53\textwidth]{Fig4} \caption{The block diagram of the real-time detection and tracking algorithms on FPGA Cyclone-V} \label{fig: F4} \end{figure} The proposed embeded system is designed that consists of a number of components. These components are image frame capturing, image processing and image displaying, etc. All components are synchronized and controlled from a control unit. A phase lock loop (PLL) generates clock signals that are contributed to each component for synchronous processing between modules in the system. \subsection{FPGA Configuration Mode Setting} Fig. \ref{fig: F5} shows the pins for configuring operation modes of the DE10-Nano board. When it is powered on, it can be configured from EPCS or HPS. The MSEL[4:0] pins are used to select the configuration scheme. It is implemented as a 6-pin DIP switch SW10 on the DE10-Nano board. \begin{figure}[h] \centering \includegraphics[width=0.53\textwidth]{Fig5} \caption{FPGA Configuration Mode Setting} \label{fig: F5} \end{figure} When the board is powered on and MSEL[4:0] set to “10010”, the FPGA is configured from EPCS, which is pre- programmed with the default code. If developers use the "Linux LXDE Desktop" SD Card image, the MSEL[4:0] needs to be set to “01010” before the board is powered on. Table \ref{tab: T2} describes modes according to MSEL (SW10) codes. \begin{table}[h!] \centering \caption{FPGA CONFIGURATION MODES} \begin{tabular}{l l l} \hline\hline MSEL [4:0] & Configuration Modes & Description \\ \hline %$\begin{array}{l l l} 10010 & AS & FPGA Configure from EPCS \\ 01010 & FPPx32/Compressi on & FPGA Configure from HPS \\ & Enabled/Fast POR & software: U-Boot, with image \\ & & stored on the SD card, like \\ & & LXDE Desktop (default) \\ 00000 & FPPx16/Compressi on & FPGA Configure from HPS \\ & Disabled/Fast POR & software: U-Boot, with image \\ & & stored on the SD Card \\ %\end{array}$ \hline\hline \end{tabular} \label{tab: T2} \end{table} % needed in second column of first page if using \IEEEpubid %\IEEEpubidadjcol \subsection{Viola-Jones Algorithm} Viola-Jone method (as shown in Fig. \ref{fig: F6}) that is employed in this paper consists of elements of Haar Cascade classifier, integral image and adaBoost algorithm. The algorithm needs a lot of positive images (images of faces) and negative images (images without faces) to train the classifier. In order to detect faces, features must be extracted. Each feature is a single value obtained by subtracting the sum of pixels under the white rectangle from the sum of pixels under the black rectangle. Haar cascades method processes on gray images. Color captured image will be converted into gray image, each pixel will have a value from 0 to 255 (depending on how gray or black it is). Adaboost, short for Adaptive Boosting, is a machine learning meta-algorithm formulated by Yoav Freund and Robert Schapire. It can be used in conjunction with many other types of learning algorithm to improve its performance. \begin{figure}[h] \centering \includegraphics[width=0.5\textwidth]{Fig6} \caption{Viola-Jones algorithm overview} \label{fig: F6} \end{figure} Haar-like features are digital image features used in object recognition. In this project, it is black and white rectangle to represent face characteristics as shown in Fig. \ref{fig: F7}. \begin{figure}[h] \centering \includegraphics[width=0.33\textwidth]{Fig7} \caption{Haar features for a face} \label{fig: F7} \end{figure} The filters including Haar features are assigned into an image in order to capture features in the face like the nose, the distance between two eyebrows, etc…. For face detection, Viola-Jones uses 4 basic features as shown in Fig. \ref{fig: F8}. \begin{figure}[h] \centering \includegraphics[width=0.43\textwidth]{Fig8} \caption{Simple features that are used by Viola-Jones} \label{fig: F8} \end{figure} Beside these basic features, more complex features including edge feature (as shown in Fig. \ref{fig: F9}), line feature (as shown in Fig. \ref{fig: F10}) and center surround feature (as shown in Fig. \ref{fig: F11}) are added in order to capture more details of the human face as well. \begin{figure}[h] \centering \includegraphics[width=0.43\textwidth]{Fig9} \caption{Edge feature} \label{fig: F9} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.53\textwidth]{Fig10} \caption{Line feature} \label{fig: F10} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.53\textwidth]{Fig11} \caption{Center surround feature} \label{fig: F11} \end{figure} \section{EXPERIMENTAL RESULTS} \subsection{Hardware Setting} The proposed system hardware is shown as in Fig. \ref{fig: F12}. The USB Blaster (JTAG cable) is utilized to connect the system to PC for programming. HDMI cable is utilized to connect the system to a monitor for display. The camera OV7670 is connected to the system by wires. \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig12} \caption{The proposed real-time face detection system} \label{fig: F12} \end{figure} Camera OV7670 resolution is scaled down to 320x240 pixels due to a large amount of data for face detection algorithms that need to be processed in order to specify detected face location in the video stream. Then a rectangle surrounding the face will be drawn. \subsection{Experimental Results} Many experiments are conducted in order to verify the system capability of detecting faces in a frame that contains up to 7 people. A face of a person, 2 faces among 3 people, and 4 faces among seven people in a frame are detected. The system can also perform more difficult tasks such as a man face wearing black glasses, mask, and drawn face. However, the system is not capable of detecting half of a face because there are not enough features of a face including eyebrows, eyes, nose and mouth. People taking part in doing experiments are tasked to appear in camera view with a distance range from 0.5m to 1.5m according to many scenarios as described in Table II. For example, there are 3 scenarios of camera distance of 0.5m, 1m and 1.5m, respectively. “N.A.” in the Table \ref{tab: T3} stands for “Not Applicable” while “X” is stands for “chosen distance”. There’re 14 scenarios (selected by X) for experiment with the combination between number of people and distance. The experiments are done in order to find out the detection ratio of the system for conclusion and evaluation (the best case, the worst case, the best distance, the worst distance, and maximum people) of the system. Note that each scenario is tested 100 times. \begin{table}[h!] \centering \caption{EXPERIMENT SCENARIOS} \begin{tabular}{c c c c c} \hline\hline Distance & 0.5m & 0.7m & 1m & 1.5m \\ \hline One person & X & N.A. & X & X \\ Two people & X & N.A. & X & X \\ Three people & X & N.A. & X & X \\ Four people & X & N.A. & X & X \\ Five people & N.A. & X & N.A. & X \\ \hline\hline \end{tabular} \label{tab: T3} \end{table} Experimental results according to scenarios are listed in Table \ref{tab: T4}. \begin{table}[h!] \centering \caption{EXPERIMENTAL RESULTS} \begin{tabular}{l c c c} \hline\hline Scenario & No. of detected faces & No. of undetected faces & Ratio \\ \hline One person & 1 & 0 & 1/1 \\ Two people & 2 & 0 & 2/2 \\ Three people & 2 & 1 & 2/3 \\ Three people & 3 & 0 & 3/3 \\ Four people & 4 & 0 & 4/4 \\ Five people & 4 & 1 & 4/5 \\ Seven people & 5 & 2 & 5/7 \\ A man & 1 & 0 & 1/1 \\ wearing glasses & & & \\ Two people & 2 & 0 & 2/2 \\ wearing glasses & 1 & 1 & 1/2 \\ Three people & 2 & 1 & 2/3 \\ wearing glasses & 1 & 2 & 1/3 \\ Four people & 2 & 2 & 2/4 \\ wearing glasses & 1 & 3 & 1/4 \\ Person wearing & 1 & 0 & 1/1 \\ mask & & & \\ Person with & 1 & 0 & 1/1 \\ a drawn face & & & \\ \hline\hline \end{tabular} \label{tab: T4} \end{table} Several results are illustrated as in from Fig. \ref{fig: F13} to Fig. \ref{fig: F17}. \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig13} \caption{One person scenario (0.5m – 1m – 1.5m)} \label{fig: F13} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig14} \caption{Two people scenario (0.5m – 1m – 1.5m)} \label{fig: F14} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig15} \caption{Three people scenario (0.5m – 1m – 1.5m)} \label{fig: F15} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig16} \caption{Four people scenario (0.5m – 1m – 1.5m)} \label{fig: F16} \end{figure} \begin{figure}[h] \centering \includegraphics[width=0.47\textwidth]{Fig17} \caption{Five people scenario (0.7m – 1.5m)} \label{fig: F17} \end{figure} All obtained experimental results in terms of detection rate and speed are summarized in Table \ref{tab: T5}. \begin{table}[h!] \centering \caption{EXPERIMENTAL RESULTS} \begin{tabular}{c c c c c} \hline\hline \multirow{2}{2em}{Scenario} & No. & Detected & Detection & Min-Max \\ & Faces & Faces & Rate (\%) & Time (ms) \\ \hline One-0.5m & 100 & 96 & 89 & 105 – 286 \\ One-1m & 100 & 84 & 78 & 132 – 487 \\ One-1.5m & 100 & 51 & 43 & 241 – 568 \\ Two-0.5m & 200 & 171 & 81.5 & 234 – 887 \\ Two-1m & 200 & 157 & 72.5 & 165 – 879 \\ Two-1.5m & 200 & 86 & 38 & 360 – 637 \\ Three-0.5m & 300 & 193 & 58.3 & 228 – 451 \\ Three-1m & 300 & 165 & 48 & 171 – 638 \\ Three-1.5m & 300 & 102 & 26.7 & 206 – 428 \\ Four-0.5m & 400 & 185 & 39.5 & 134 – 443 \\ Four-1m & 400 & 169 & 35 & 179 – 379 \\ Four-1.5m & 400 & 114 & 20.5 & 280 – 336 \\ Five-0.7m & 500 & 216 & 36.6 & 178 – 349 \\ Five-1.5m & 500 & 173 & 27.4 & 187 - 581 \\ \hline\hline \end{tabular} \label{tab: T5} \end{table} According to Table \ref{tab: T5}, the proposed system works on the optimal performance at the distance 0.5m – 1m and 1 - 2 people. Detection rate of the system is evaluated by each case. It is not likely to combine cases in order to calculate a common value because detection rate will be affected by the detection distance and the number of people. Maximum people that the system is capable of detecting is 5 with a low detection rate. In the case of one person, we tried testing to find out the maximum distance which the system is capable of detecting faces. That maximum distance is 1.8m. Minimum time the system is able to detect faces is 105ms. Maximum time the system is capable of detecting faces is 887ms. In addition, several experiments were done in order to evaluate the proposed system performance to other existing systems such as OpenCV and MATLAB Tools for face detection in terms of processing time (speed) and detection rate. A set of images (484 images containing 641 faces) were used to perform measurements. Table \ref{tab: T6} shows the comparison of the proposed hardware and software tool (OpenCV) in terms of detection rate. The results show that the software tool is more accurate than the proposed hardware. \begin{table}[h!] \centering \caption{COMPARISON BETWEEN HARDWARE AND SOFTWARE IN TERMS OF DETECTION RATE WITHIN 05.M DISTANCE OF FACE AND CAMERA} \begin{tabular}{c c c} \hline\hline Tool & Detected Faces & Detection Rate (\%)\\ \hline Hardware on DE10-Nano & 590 & 92 \\ OpenCV on Intel 7 CPU & 619 & 97 \\ \hline\hline \end{tabular} \label{tab: T6} \end{table} Above set of images were repeatedly used for another experiment with the same purpose but in terms of speed. Table \ref{tab: T7} shows the comparison of the proposed hardware and software tool (MATLAB) in terms of processing time (speed). The results show that the proposed hardware is faster than the MATLAB tool because of the parallel architecture of the processor. \begin{table}[h!] \centering \caption{COMPARISON BETWEEN HARDWARE AND SOFTWARE IN TERMS OF SPEED WITHIN 05.M DISTANCE OF FACE AND CAMERA} \begin{tabular}{c c c} \hline\hline Tool & Image Nature & Time (ms)\\ \hline \multirow{2}{3em}{Hardware} & Image with single face & 113.3 \\ & Image with multiple face & 134.0 \\ \multirow{2}{3em}{Software} & Image with single face & 1734 \\ & Image with multiple face & 2289 \\ \hline\hline \end{tabular} \label{tab: T7} \end{table} After doing all the experiments, it is concluded that the software face detection system is capable of processing the images at speeds of an average of 0.71 fps with 320x240 pixel images and 0.37 fps with 640x480 pixel images. Moreover, the hardware face detection system has the performance improvement up to 37.33 times faster than the software face detection system with 320x240 pixel images and up to 18.8 times faster than the software face detection system with 640x480 pixel image. Table \ref{tab: T8} shows the performance with 320x240 resolution images while Table \ref{tab: T9} shows the performance with 640x480 resolution images. \begin{table}[h!] \centering \caption{RESULT PERFORMED WITH 320X240 RESOLUTION IMAGES} \begin{tabular}{c c c} \hline\hline Total Faces & Software & Hardware \\ \hline 1 & 1.256 ms (0.79 fps) & 34.712 ms (28.80 fps) \\ 6 & 1.402 ms (0.71 fps) & 37.378 ms (26.75 fps) \\ 11 & 1.538 ms (0.65 fps) & 41.711 ms (23.97 fps) \\ \hline\hline \end{tabular} \label{tab: T8} \end{table} \begin{table}[h!] \centering \caption{RESULT PERFORMED WITH 640X480 RESOLUTION IMAGES} \begin{tabular}{c c c} \hline\hline Total Faces & Software & Hardware \\ \hline 1 & 2.165 ms (0.46 fps) & 133.143 ms (7.51 fps) \\ 6 & 2.919 ms (0.34 fps) & 146.745 ms (6.81 fps) \\ 11 & 3.129 ms (0.31 fps) & 152.664 ms (6.55 fps) \\ \hline\hline \end{tabular} \label{tab: T9} \end{table} % An example of a floating figure using the graphicx package. % Note that \label must occur AFTER (or within) \caption. % For figures, \caption should occur after the \includegraphics. % Note that IEEEtran v1.7 and later has special internal code that % is designed to preserve the operation of \label within \caption % even when the captionsoff option is in effect. However, because % of issues like this, it may be the safest practice to put all your % \label just after \caption rather than within \caption{}. % % Reminder: the "draftcls" or "draftclsnofoot", not "draft", class % option should be used if it is desired that the figures are to be % displayed while in draft mode. % %\begin{figure}[!t] %\centering %\includegraphics[width=2.5in]{myfigure} % where an .eps filename suffix will be assumed under latex, % and a .pdf suffix will be assumed for pdflatex; or what has been declared % via \DeclareGraphicsExtensions. %\caption{Simulation results for the network.} %\label{fig_sim} %\end{figure} % Note that the IEEE typically puts floats only at the top, even when this % results in a large percentage of a column being occupied by floats. % An example of a double column floating figure using two subfigures. % (The subfig.sty package must be loaded for this to work.) % The subfigure \label commands are set within each subfloat command, % and the \label for the overall figure must come after \caption. % \hfil is used as a separator to get equal spacing. % Watch out that the combined width of all the subfigures on a % line do not exceed the text width or a line break will occur. % %\begin{figure*}[!t] %\centering %\subfloat[Case I]{\includegraphics[width=2.5in]{box}% %\label{fig_first_case}} %\hfil %\subfloat[Case II]{\includegraphics[width=2.5in]{box}% %\label{fig_second_case}} %\caption{Simulation results for the network.} %\label{fig_sim} %\end{figure*} % % Note that often IEEE papers with subfigures do not employ subfigure % captions (using the optional argument to \subfloat[]), but instead will % reference/describe all of them (a), (b), etc., within the main caption. % Be aware that for subfig.sty to generate the (a), (b), etc., subfigure % labels, the optional argument to \subfloat must be present. If a % subcaption is not desired, just leave its contents blank, % e.g., \subfloat[]. % An example of a floating table. Note that, for IEEE style tables, the % \caption command should come BEFORE the table and, given that table % captions serve much like titles, are usually capitalized except for words % such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to % and up, which are usually not capitalized unless they are the first or % last word of the caption. Table text will default to \footnotesize as % the IEEE normally uses this smaller font for tables. % The \label must come after \caption as always. % %\begin{table}[!t] %% increase table row spacing, adjust to taste %\renewcommand{\arraystretch}{1.3} % if using array.sty, it might be a good idea to tweak the value of % \extrarowheight as needed to properly center the text within the cells %\caption{An Example of a Table} %\label{table_example} %\centering %% Some packages, such as MDW tools, offer better commands for making tables %% than the plain LaTeX2e tabular which is used here. %\begin{tabular}{|c||c|} %\hline %One & Two\\ %\hline %Three & Four\\ %\hline %\end{tabular} %\end{table} % Note that the IEEE does not put floats in the very first column % - or typically anywhere on the first page for that matter. Also, % in-text middle ("here") positioning is typically not used, but it % is allowed and encouraged for Computer Society conferences (but % not Computer Society journals). Most IEEE journals/conferences use % top floats exclusively. % Note that, LaTeX2e, unlike IEEE journals/conferences, places % footnotes above bottom floats. This can be corrected via the % \fnbelowfloat command of the stfloats package. \section{Conclusion} In this paper, a real-time face detection and human tracking system that utilizes the fpga DE10-NANO controller board with Viola-Jones algorithm embedded is proposed. The project is carried out on Quartus tool for data transmission with 2 main functions: from camera to the board and from the board output to the screen. The proposed system performs detection rate up to 92\% and the fastest processing time 113.3 ms. The proposed system is not as accurate as the software tool, but it is flexible, easy to install and use because there is only one controller board. Moreover, the proposed system gives results much faster than the software tool. % if have a single appendix: %\appendix[Proof of the Zonklar Equations] % or %\appendix % for no appendix heading % do not use \section anymore after \appendix, only \section* % is possibly needed % use appendices with more than one appendix % then use \section to start each appendix % you must declare a \section before using any % \subsection or using \label (\appendices by itself % starts a section numbered zero.) % %\appendices %\section{Proof of the First Zonklar Equation} %Appendix one text goes here. % %% you can choose not to have a title for an appendix %% if you want by leaving the argument blank %\section{} %Appendix two text goes here. % use section* for acknowledgment \section*{Acknowledgment} The authors would like to thank to students PHAN TRUONG KHANG and LA NGOC LE from the Department of Computer Engineering for implementing and testing the proposed system % Can use something like this to put references on a page % by themselves when using endfloat and the captionsoff option. \ifCLASSOPTIONcaptionsoff \newpage \fi % trigger a \newpage just before the given reference % number - used to balance the columns on the last page % adjust value as needed - may need to be readjusted if % the document is modified later %\IEEEtriggeratref{8} % The "triggered" command can be changed if desired: %\IEEEtriggercmd{\enlargethispage{-5in}} % references section % can use a bibliography generated by BibTeX as a .bbl file % BibTeX documentation can be easily obtained at: % http://mirror.ctan.org/biblio/bibtex/contrib/doc/ % The IEEEtran BibTeX style support page is at: % http://www.michaelshell.org/tex/ieeetran/bibtex/ %\bibliographystyle{IEEEtran} % argument is your BibTeX string definitions and bibliography database(s) %\bibliography{IEEEabrv,../bib/paper} % % manually copy in the resultant .bbl file % set second argument of \begin to the number of references % (used to reserve space for the reference number labels box) \begin{thebibliography}{1} \bibitem{IEEEhowto:kopka} Yu Wei, Xiong Bing and C. Chareonsak, "FPGA implementation of AdaBoost algorithm for detection of face biometrics," IEEE International Workshop on Biomedical Circuits and Systems, 2004., 2004, pp. S1/6-17, doi: 10.1109/BIOCAS.2004.1454161. \bibitem{IEEEhowto:kopka} P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 2001, pp. I-I, doi: 10.1109/CVPR.2001.990517. \bibitem{IEEEhowto:kopka} Nair, V., Laprise, PO. and Clark, J.J. An FPGA-Based People Detection System. EURASIP J. Adv. Signal Process. 2005, 272174 (2005). https://doi.org/10.1155/ASP.2005.1047 \bibitem{IEEEhowto:kopka} Hichem, B.F., Ahmed, E., and Ben, J., "An Efficient and Flexible FPGA Implementation of a Face Detection System," Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate ArraysFebruary 2015 Pages 261https://doi.org/10.1145/2684746.2689095 \bibitem{IEEEhowto:kopka} R. F. Rahmat, Y. Azzakirot and T. Z. Lini, "Tree Identification to Calculate the Amount of Palm Trees using Haar-Cascade Classifier Algorithm," 2019 3rd International Conference on Electrical, Telecommunication and Computer Engineering (ELTICOM), 2019, pp. 36-39, doi: 10.1109/ELTICOM47379.2019.8943897. \bibitem{IEEEhowto:kopka} S. Choudhury, S. P. Chattopadhyay and T. K. Hazra, "Vehicle detection and counting using haar feature-based classifier," 2017 8th Annual Industrial Automation and Electromechanical Engineering Conference (IEMECON), 2017, pp. 106-109, doi: 10.1109/IEMECON.2017.8079571. \bibitem{IEEEhowto:kopka} A. AbdelRaouf and D. Salama, "Handwritten Signature Verification using Haar Cascade Classifier Approach," 2018 13th International Conference on Computer Engineering and Systems (ICCES), 2018, pp. 319-326, doi: 10.1109/ICCES.2018.8639437. \bibitem{IEEEhowto:kopka} Y. Wang, L. Li, X. Wan and J. Wang, "Woven Fabric Defect Detection Based on the Cascade Classifier," 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 2019, pp. 1-5, doi: 10.1109/CISP-BMEI48845.2019.8965739. \bibitem{IEEEhowto:kopka} Terasic Inc, DE10-Nano User manual, 2003-2017, last accessed 2019/10/27 \end{thebibliography} % biography section % % If you have an EPS/PDF photo (graphicx package needed) extra braces are % needed around the contents of the optional argument to biography to prevent % the LaTeX parser from getting confused when it sees the complicated % \includegraphics command within an optional argument. (You could create % your own custom macro containing the \includegraphics command to make things % simpler here.) %\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{mshell}}]{Michael Shell} % or if you just want to reserve a space for a photo: \begin{IEEEbiography}{Huu Luong NGUYEN} Biography text here. \end{IEEEbiography} % if you will not have a photo at all: \begin{IEEEbiography}{Minh Son NGUYEN} Biography text here. \end{IEEEbiography} % insert where needed to balance the two columns on the last page with % biographies %\newpage \begin{IEEEbiography}{Tri Nhut DO} Biography text here. \end{IEEEbiography} % You can push biographies down or up by placing % a \vfill before or after them. The appropriate % use of \vfill depends on what kind of text is % on the last page and whether or not the columns % are being equalized. %\vfill % Can be used to pull up biographies so that the bottom of the last one % is flush with the other column. %\enlargethispage{-5in} % that's all folks \end{document}