Looking at big brother applications: Homeland security, Patriot Act, industry consultation or operation dystopia

Introduction To date, vendors serving the video surveillance industry have used analog CCTV cameras and interfaces as the basis for their surveillance systems. These system components are not easily expandable, and have low video resolution with little or no signal processing. However, the next generation of video surveillance systems will replace these components with newer digital LAN cameras, complex image processing, and video-over-IP routing. They will no longer be simply surveillance camera systems, but also video communication systems.

The Internet Protocol (IP)-based structure of these new surveillance systems allows for scalability, flexibility, and cyber security. Various encoding and decoding standards transport the video stream with MPEG4 CODEC being the standard used today for compressing and decompressing data. Besides the CODEC function, image pre- and postprocessing enhances the picture quality in real time with low latency. Programmable logic with embedded DSP blocks, memories, interfaces, and off-the-shelf intellectual property solutions allows a designer to meet the new digital system requirements.

Digital video recorder architecture In a Digital Video Recorder (DVR) system, multiple analog CCTV cameras route to a central video switching hub for storage, scaling, image processing and display. Video resolution and quality are typically low to reduce complex compression and cost. Special processing such as motion detection reduces the amount of storage space in the central hub by only capturing video when movement occurs. This architecture is therefore not flexible or readily expandable, so video monitoring is limited in terms of quality and quantity.

A typical DVR system (shown in Figure 1) combines with either an internal or external video matrix switcher to route the video from cameras to monitors. This type of system requires multiple inputs and output multiplexing, making it very suitable for using programmable logic for system flexibility and expandability.

Figure 1

Video compression and image processing There are many different standards for video data compression, with the most popular including JPEG, H.263, Motion JPEG, MPEG and Wavelet. The type of compression used has an impact on hardware system requirements, including memory, data rate, and storage space. Next-generation surveillance systems will probably use the H.264 standard due to its compression efficiency. Efficiency is a key factor in the transmission of high-quality video over a bandwidth-limited network. For example, a color transmission at 30 fps at 640 x 480 pixels requires a high data rate of 26 Mbps. This data rate must be brought down (compressed) to a more manageable data rate that can be routed over a twisted pair of copper wires.

The two types of video compression data rate are constant bit rate (CBR) and variable bit rate (VBR). CBR limits the data rate for a real-time communication channel with limited bandwidth. However, when CBR compresses high motion details, image quality is lost and results in image blocks on the display. VBR allows the data rate to adapt to the motion or absence of motion. This is extremely useful for video surveillance system storage. H.264 compression with VBR provides the best efficiency for security video storage.

Pre- and postprocessing techniques, such as de-interlacing, scaling, noise reduction using 2D filtering, and color space conversion, are also critical parts of a video surveillance system. Figure 2 illustrates a typical video surveillance system setup using an FPGA and an application specific standard product (ASSP).

Figure 2
(Click graphic to zoom by 1.3x)

With expanding resolutions and evolving compression, there is a need for high performance while keeping architectures flexible to allow for quick upgradeability. As technology matures and volumes increase, the focus will move to cost reduction. System architecture choices include standard cell ASICs, ASSPs, and programmable solutions such as digital signal processing (DSP) or media processors and FPGAs. Each of the approaches has advantages and disadvantages, with the ultimate choice depending on end-equipment requirements and solution availability. The ideal surveillance architecture should have the following characteristics: high performance, flexibility, easy upgradeability, low development cost, and a migration path to lower unit costs as the application matures and volume ramps.

High performance Performance not only applies to compression, but also pre- and post-processing functions. In fact, in many cases these functions consume more performance than the compression algorithm itself. Examples of these functions include scaling, de-interlacing, filtering, and color space conversion.

For video surveillance, the need for high performance rules out processor-only architectures. They simply cannot meet the performance requirements with a single device. A state-of-the-art DSP running at 1 GHz cannot perform H.264 HD decoding, and H.264 HD encoding is approximately ten times more complex than decoding. FPGAs are the only programmable solutions able to tackle this performance problem based on their inherent design characteristics. In some cases, the best solution is a combination of an FPGA plus an external DSP processor.

Flexibility provides fast time to market and easy upgradeability When technology rapidly evolves, architectures must be flexible and easy to upgrade. This rules out standard-cell ASICs and ASSPs for those applications. Typically designed for very high volume consumer markets, ASSPs often are quickly obsolete, making them an extremely risky choice for most applications.

Low development cost When adding up costs for masks and wafers, software, design verification, and layout, development of a typical 90-nm standard-cell ASIC can cost as much as $30 million. Only the highest volume consumer markets can justify such pricey development costs. Such ASICs require thirty or more precision layers to create the various circuits and interconnect lines, make up the bulk of the NRE costs commonly quoted in an ASIC design. These charges can be spent again if a design revision occurs for any reason. Today's wafer foundries make wafers with diameters of 300mm (11.8") and prefer a minimum-order production run that uses 25 of these wafers. By way of example, a modest sized chip design that is 1cm on a side yields about 500 die per wafer, so a minimum order quantity results in 12,500 chips. One minimum order run can sometimes be a lifetime supply of chips for many low-volume applications.

Migration path to lower unit costs As standards stabilize and volumes increase, it is important to have a solution with a low-cost migration path. Often this means either market-focused ASSPs or standard-cell custom ASIC devices. However, the rising cost of custom silicon makes those solutions economically feasible in only the highest volume consumer applications. Most silicon companies with a focus on video and imaging target applications such as video camcorders, set-top boxes, digital still cameras, cell phones and other portable products, or LCD televisions and monitors. When designing a lower-volume type of application, considering an FPGA is likely the wisest decision, as it is unlikely an ASSP with the exact feature set required exists and even the best off-the-shelf solution is a high-risk choice due to the potential for obsolescence.

Video and image processing solutions FPGAs are particularly well suited to meet the requirements of video surveillance and as well as many other video and image processing applications. FPGAs having the following characteristics are very appealing for video and image processing architectures:

  • High performance: High definition (HD) processing can be implemented in a single FPGA.
  • Flexibility: FPGAs provide the ability to upgrade architectures quickly to meet evolving requirements, while scalability allows use of FPGAs in low-cost and high-performance systems.
  • Low development cost: Video development kits that include the software tools required to develop a video system using FPGAs are available and low cost.
  • Obsolescence proof: FPGAs have a very large customer base that ship products for many years after introduction. Also, FPGA designs are easily migrated from one process node to the next.
  • Structured ASIC migration path to low costs: structured ASICs, that is HardCopy structured ASICs from Altera start at $15 at 100ku for 1 million ASIC gates.

ASSP-like functionality on FPGAs and structured ASICs FPGAs or structured ASICs are examples of the growing number of solutions with ASSP functionality that are available to designers for video surveillance applications. An example is the H.264 Main Profile Standard Definition Encoder from ATEME SA, a French-based provider of advanced video compression technology products. With this product, customers are using FPGAs just as they would an ASSP. The benefit over the traditional ASSP approach is that the FPGA solution evolves quickly, with no risk of obsolescence.

DSP design flow For custom development, an optimized DSP design flow that allows several different ways to represent the design includes VHDL/Verilog, model-based design and C-based design. Intellectual property cores can be used in conjunction with any of these design flow options.

Companies are partnering to create comprehensive DSP development flows enabling designers to enjoy the price/performance benefits FPGAs offer when leveraging model-based design tools such as Simulink. DSP development tools such as DSP Builder connect Simulink with Quartus II development software. DSP Builder provides a seamless design flow in which designers perform algorithmic development in MATLAB software and system-level design in Simulink software, and then port the design to hardware description language (HDL) files for use in Quartus II software. The DSP Builder tool is tightly integrated with the Altera SOPC Builder tool, allowing the user to build systems that incorporate Simulink designs and Nios II embedded processor and intellectual property cores. This development flow is easy and intuitive for designers who do not have extensive experience using programmable logic design software.

Video and image processing functions Integrated functionality that can be optimized for specific applications is important when designing new video surveillance solutions. Table 1 summarizes these functions.




Converts interlaced video formats to progressive video format

Color Space Converter

Converts image data between a variety of different color spaces


Resizes and clips image frames

Alpha Blending Mixer

Mixes and blends multiple image streams

Gamma Corrector

Performs gamma correction on a color plane/space

Chroma Resampler

Changes the sampling rate of the chroma data for image frames

2D Filter

Implements finite impulse response (FIR) filter operation on an image-data stream to smooth or sharpen images

2D Median Filter

Removes noise in an image, replacing each pixel value with the median of neighboring pixel values

Line Buffer Compiler

Efficiently maps image line buffers to on-chip memories

Table 1

The 2D Filter GUI is shown in Figure 3 as an example of the type of user configuration available with the cores provided in an integrated video and image processing suite. Resolutions, bits per sample, FIR filter size, edge behavior, overflow behavior, and accumulator length are all static parameters supported in the 2D filter core.

Figure 3

Video compression Several third parties have video compression solutions targeting FPGAs and structured ASICs. Some of the common video compression standards that today’s designers must accommodate in their applications are H.264 and High Profile, H.264 Baseline Profile, JPEG/JPEG2000, and MPEG4 SP/ASP. These standards are ideally targeted by using FPGAs.

Video over IP A reference design for video over IP applications is shown in Figure 3. The design transmits video transport stream (TS) data over IP-based networks, bridging one or more compressed video streams and IP packets over 100-Mbps or 1-Gbps Ethernet. The reference design accepts TS data and encapsulates it for transmission over Ethernet, as well as receiving frames from Ethernet and generating TS data.

Encapsulation of the TS data for Ethernet uses IP and the user datagram protocol (UDP). The real-time transport protocol (RTP) can also optionally be used. Dedicated hardware performs the encapsulation, maximizing the throughput of the reference design and minimizing latency. Frames can be processed, transmitted and received at the Ethernet line rate, which supports an aggregate TS bandwidth of over 900 Mbps for a gigabit Ethernet link. For multiple TS interfaces, the reference design individually maps each one to a specific UDP/IP socket (combination of IP address and UDP port). All other encapsulation parameters can also be individually configured per TS. The reference design supports IP multicast and includes a Nios II processor. Software running on the Nios II processor configures the operation of the reference design and handles any Ethernet management traffic.

Conclusion The ideal surveillance architecture should have the following characteristics: high performance, flexibility, easy upgradeability, low development cost, and a migration path to lower cost as the application matures and volume ramps. FPGAs in conjunction with the feature-rich video image libraries, video over IP reference design, and compression solutions offer video system designers all the key building blocks needed to produce a system.

FPGAs are an ideal fit for video surveillance applications where there is a need to have a programmable solution that is inherently scalable for improving cost, performance, flexibility and productivity requirements while meeting time-to-market goals. FPGAs are a logical design choice to address the market trends for video surveillance applications.