Multi-core architecture of a digital Camera for increasing performance

Research Area: Multi-core for Real-time Embedded Applications
Authors: Bharathram. P, Rajshekar. K, Saketh Paranjape, Sowmya. P
Faculty mentor: Ashok Kumar
Name of the Institution: B.M.S.College of Engineering

___________________________________________________________________________________________________________________

Abstract:

Traditionally digital cameras are designed on a single core architecture which performs and controls all the tasks. This however is subject to certain limitations which can be overcome by taking advantage of the inherent parallelism among the tasks. In an attempt to achieve real-time capture we present multi-core camera architecture, propose methods for implementing the same and resolve the prevailing issues.

Introduction:

In recent years, digital camera products have grown explosively as digital photos and videos have become more and more popular in our daily life. Thanks to technology innovations, digital cameras are used in a variety of applications, such as digital still cameras, camcorders, camera phones and video surveillance.

Digital camera is a complex opto-electronics system composed by lens motors and digital circuits. Etc. Digital camera processor is the core of a digital camera. A typical digital camera system based on a single-chip processor is illustrated in Figure 1.
Fig 1: Architecture of a single core digital camera [1]

In a digital camera, the pictures are taken in many steps. Firstly images captured by charge-coupled device (CCD) through optical lens are transformed to digital format by A/D converter. Then the digital image signals are interpolated and enhanced to raster pictures. The pictures are then either sub-sampled to display on the screen of LCD or TV, or compressed into JPEG format to be saved in the nonvolatile cards such as compact flash or smart media. The stored pictures can be either decompressed to playback or transferred into connectors through USB or EPP ports.

In this case the digital camera processor performs various tasks such as digital image processing, JPEG encoding, data storage, data transfer. It controls light setting, AWB (auto white balance) and AF (auto focus) through process intensive algorithms, scheduling of which on multiple cores shall be the main focus of this paper.

The various scene setup tasks:

Auto focus (AF): Auto focus determines the distance to the subject by computer analysis of the image itself. The camera actually looks at the scene and drives the lens back and forth searching for the best focus (Image is of highest sharpness).
Light setting: Auto exposure comprises of adjusting the aperture size, shutter speed and determining whether flash is to be used.
Smile detection: The software looks for the open eyes and upturned mouth associated with a smile once it identifies a face or faces on the screen. When smile detection is enabled, the camera chooses the moment to take a picture after the user has composed the shot.

The post processing involves color balancing. The process of color balancing includes Black level adjustment, Noise reduction, White balance, CFA Interpolation, color correction, Gamma correction, edge enhancement and contrast enhancement.

Color balance changes the overall mixture of colors in an image and is used for color correction; generalized versions of color balance are used to get colors other than neutrals to also appear correct or pleasing.

Fig 2: Sequence of events during post-processing [2]

The camera is usually operated in two modes. Single shot mode and multi shot mode. Single shot mode wherein one picture is taken at a time. For each picture the scene is setup completely thereby ensuring high quality of image.Multi-shot (burst) Mode: In this mode, users can take multiple pictures. The scene is setup once or partially each time, resulting in reduced quality of image.

Problem Statement:

Auto-focus lags:

Consider a scenario where it is necessary to capture the scene of a bird pecking on a branch. Say the camera is in auto mode, initially there is a lag before the focus is set on the object in the scene. Once the focus is set, the camera is ready to capture the scene.

At the instance of shutter release (just when the shutter release button is on the verge of being fully pressed) the bird readies itself to depart from the branch. As the bird departs there is a change in the position of the object. This change upsets the lens' focus. Running the auto focus algorithm is usually not feasible at this instance. The camera goes ahead with its task of image capture with a distorted focus. By the time the scene is captured taking into account the shutter release lag, the object of interest would have been rendered out of focus in the resulting image, violating the idea of real time capture.

Burst mode performance issues:

In most conventional cameras there are serious performance issues when it comes to shooting in the burst mode. In the burst mode the camera takes a series of images in fast succession until the button is released. In the burst mode the camera has to perform a series of tasks sequentially and repeatedly in a short burst. The Load on the processor is much higher than the load it handles while shooting in the single shot mode. The problems that plague the conventional cameras in burst mode are regarding the number of frames shot per second , the delay in between shots , the inability of the camera to adjust to constantly changing light factors, time taken for image processing , sluggish response and blanking of the LCD display. Identifying inherent parallelism among the tasks and scheduling them in a parallel manner, the degree of slack in camera's performance diminishes extensively.

Methodology:

The proposed camera architecture is as follows:

The main processing unit consists of a multi-core processor instead of the conventional single core. We shall consider dual core architecture for our discussion. We shall refer to these cores as Core A and Core B. These two cores control all the processes in the camera. However the two cores operate as peers. The control can be direct by setting up parameters through the use of DACs as in the case of aperture setting, or by issuing signals to a hard-wired processing unit which in turn performs the required operations. The algorithms used for the various processes are the conventional algorithms applicable to the single core processor.

We shall be considering the automatic mode of operation of the camera for our discussion here as the tasks performed by the camera in this mode is the maximum. But looking at manual or programmable modes of operation of the camera, we realize that the tasks performed by the camera are not very process intensive. Hence we can deactivate one of the processing cores to save power, thereby increasing the battery life of the camera for an advanced user.

The tasks of handling the user interface and monitoring the system are handled on core A and core B respectively, at all times.

Fig 3: Proposed dual core architecture

We shall consider the two common modes of operation of a camera, Single shot and Multi shot (Burst mode), to understand the working of the system.
Functioning of a single shot mode in the multi core architecture:

Fig 4:

The tasks comprising the setting of the scene in the order of processing intensity are:

1. Auto focus
2. Smile detection
3. Light setting.

The above tasks are mutually independent, and only read the contents of the buffer and do not manipulate it.
The auto focus process runs on core A while Smile detection and light setting processes run on core B. The processes are so grouped to enable a near balanced CPU Utilization.

In this state, the auto focus constantly running on core A maintains focus on the object at all times. While simultaneously core B monitors the scene, adjusts the lighting parameter and executing the smile detection algorithm. This facilitates a good quality of image, with the least lag between clicking and capture of the image, hence making the camera highly responsive overcoming the auto-focus lag discussed earlier.

After capturing the image the post processing operations, that is noise reduction, color balance, red-eye correction, are performed on the image followed by an initiation of DMA for storage. These tasks are performed on Core B while AF continues to run on Core A. As a result a part of the pre-processing tasks are already performed while the post-processing is being performed. Therefore the time requirement for setup of the next scene is reduced.

Fig 5: Burst mode:

At any point while taking multiple photographs, one core is setting up the scene while the other is performing post processing on the previously captured image. The two cores operate in tandem. Signals are issued to maintain synchronization between the two processors.
This prevents the processors from simultaneously driving the motor, charging the CCD and setting up the flash also prevents the processors from simultaneously writing to secondary storage.

This way the lag between the shots is reduced as compared to a single core camera system, thereby increasing the number of pictures per unit time. Also, most single core digital cameras don’t provide a display in burst mode operation due scheduling constraints on a single core. However in this multi-core system the scheduling constraints are overcome and the display of the burst photographs can be displayed.

Fig 6:Flowchart depicting functioning of burst-mode operation.

Key results:

1.      Reduced time taken for setting up the scene.
2.      Reduced delay between clicking and capture of the image.
3.      Increased number of pictures per unit time in burst mode option.
4.      Increased quality of image in both operational modes, as the pre-processing is near real-time.
5.      Capability of executing complex algorithms exploiting parallelism.

Discussion:

The advantages of this system by the virtue of the use of a multiple core architecture shall be discussed here.The power requirement of a multiple core system is lesser as compared to a single core system; this is due to the fact that the power consumption of a processor does not vary linearly with respect to the clock speed. Hence the load on the battery is minimized.

The Multi-core architecture provides for parallel processing capability. This can be to perform independent tasks simultaneously (which we will be topic of discussion of our paper), or exploit inherent parallelism in a process by running the process on the two cores.

The increased processing capability can provide for a more complex user interface system to perform manual post processing of the images by the users. A system such as panoramic photo-stitching or photo editing applications can be incorporated into the system.

The effects of this architecture on problems discussed earlier are as follows:

When the position of the object changes, the corresponding change in the focus is accommodated by the camera as a direct result of running the auto-focus algorithm continuously on a separate core. Consequently the object falls within better focus and the image is pristine.
In the burst mode, the delay between the shots is reduced to a certain degree as the tasks are scheduled to run in parallel, cyclical on the 2 cores. Since, the user interface program is not process intensive; it does not load the processors, thereby overcoming the problem of blanking of LCD display.

On the flip side the manufacturing costs of a multiple core system is higher. The dual-core system proposed does not decrease processing time two fold, as expected ideally.

Conclusion:

The architecture discussed increases performance and quality of image captured with power consumption comparable to that of the single core architecture. This is however achieved without changing the algorithms that are currently being implemented.

Scope for the future:

The various post processing tasks mentioned above are sequential in nature. Determining inherent parallelism in the nature of working and developing algorithms to exploit the same has scope for further development. Photo editing programs of high complexity may be incorporated in to the system, thus allowing the user to edit photographs on the camera itself.

_______________________________________________________________________________________________________________________________________

References:

1.       NEC electronics. www.necel.com
2.       [IEEE] IMAGE PIPELINE TUNING FOR DIGITAL CAMERAS by Jianping Zhou and John Glotzbach,Texas Instruments
3.       [IEEE] System-on-Chip for Mega-Pixel Digital Camera Processor with Auto Control Functions: ASIC & System State Key Laboratory, Micro electronics department, Fudan University, Shanghai.

Acknowledgement:

First and foremost we would like to thank Mr. Ashok Kumar for his guidance and insight without which this paper would not have been possible. We would like to thank the B.M.S.C.E data center staff for co-operation and support.