Eye of the beholder: FPGAs convert CMOS imager output to human-viewable RGB

Paul describes using the Altera Cyclone III FPGA in an application to render a CMOS imager’s output human-viewable.

CMOS imager Bayer-pattern pixel data conversion to RGB pixel data can be readily accomplished using today’s low-cost FPGA technology devices. Pixels within modern CMOS image sensors are typically arranged in a Bayer color pattern. Figure 1 presents an example of that pattern.

Figure 1: Modern CMOS image sensors arrange pixels in Bayer color pattern.

Image data is scanned from left to right and from top to bottom. Each scanned raw pixel is a monochrome color element, either Red or Green or Blue. Inherent characteristics of the human eye make it advantageous to have twice as many green pixels as red or blue ones.

In order to render a CMOS imager’s output human-viewable, a conversion from the Bayer image data format to an RGB image data format is necessary. A common method of converting Bayer image data to RGB data is the bilinear-interpolation method in which a 3 x 3 Bayer-pixel matrix is mathematically combined into a single pixel with separate Red, Green, and Blue data component values. Thus a 3x3 matrix of monochrome pixels is transformed into a single pixel with three component values. The math for bilinear interpolation (as well as algorithms for other image conversion methods) may be readily found in industry text books.

At Orchid Technologies Engineering and Consulting, Inc. we implemented the bilinear interpolation based image conversion using an Altera Cyclone III FPGA. We selected the Cyclone III FPGA device for its low-cost, resource-rich architecture.

We began our FPGA implementation by dividing the complex function up into seven individual building blocks. These blocks were:

n  Input Data Formatter

n  Image Line Data Memory

n  Image Line Data Selector          

n  Bayer Matrix Calculator

n  Output Data Formatter

n  PLL Clocking and Control

n  Built-in Test Pattern Generator

Working with Altera Corporation’s Quartus II development platform we implemented and tested each subsystem. Our goal was the development of a 10-bit wide conversion system that preserved data width throughout the process. Preservation of color depth and resolution was specifically necessary for our application. Many low-cost commercial imagers will provide 16-bit wide RGB 565 data, YUV 4:2:2 data, or some other truncated color depth output. Unique to our conversion approach was the 30-bit wide color depth with 10 bits per pixel per color. Our approach was designed for a maximum line width of 1024 pixels at a maximum input pixel rate of 27 MHz.

Our Cyclone III implementation required 1 PLL Block, 462 logic cells, 219 logic registers, 81920 internal memory bits, and 40 I/O pins. Implemented in an EP3C5E144 device, our resource usage was under 10 percent for logic element utilization and under 20 percent for memory bit utilization. Clearly, plenty of resources remain for the implementation of other complex system features. Our image processor was an easy fit for the Cyclone III device.

Paul Nickelsberg, President and CTO of Orchid Technologies Engineering and Consulting, Inc., has over 20 years experience in electronic products design.

Orchid Technologies Engineering and Consulting, Inc.