Rockchip-isp1

From Rockchip open source Document
Revision as of 16:02, 8 January 2018 by Jacob (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

 

Overview

The document below provide basical informations about the rockchip-isp1 driver driver and Image Signal Processing block on Rockchip SoC with examples and details.

Hardware

More detailed information could be found in TRM chapter "Image Signal Processing", "MIPI D-PHY" , but they are only available under NDA.

ISP Details

ISP comprises with:

  • MIPI serial camera interface
  • Image Signal Processing
  • Many Image Enhancement Blocks
  • Crop
  • Resize

Block diagram

The completed block diagram can not be pasted from datasheet to here, below diagram is an abstract version:

Blocks.png

 

MIPI Details

There are three D-PHY instances in rockchip SoC, their connection are shown as following figure:

Phy.png

Software

Driver

rockchip-isp1 is a V4L2 based driver for Image Signal Processing block on Rockchip SoC. Compared to the earlier driver rk-isp10, it use Media Controller framework, which it's different from plain v4L2. While the plain v4L2 had a view of the device as a plain DMA based image drabber which connects the input data to the host memory, the Media Controller takes into consideration the fact that a typical video device might consist of multiple sub-devices.

Media Controller Basics

Please read below link carefully, especially if you don't have use it before.

https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.html

Block diagram

File view

Code.png

V4l2 view

Rockchip-isp1-v4l2.png

Name Type Description
rkisp1_mainpath v4l2_vdev, capture

Format: YUV, RAW Bayer; Max resolution: 4416*3312; Support: Crop

rkisp1_selfpath v4l2_vdev, capture

Format: YUV, RGB; Max resolution: 1920*1080; Support: Crop

rkisp1-isp-subdev v4l2_subdev

Internal isp blocks; Support: source/sink pad crop.

The format on sink pad should be equal to sensor input format, the size should be equal/less than sensor input size.

The format on source pad should be equal to vdev output format if output format is raw bayer, otherwise it should be  YUYV2X8. The size should be equal/less than sink pad size.

rockchip-sy-mipi-dphy v4l2_subdev

MIPI-DPHY Configure

rkisp1-statistics v4l2_vdev, capture

Provide Image color Statistics information.

rkisp1-input-params v4l2_vdev, output

Accept params for AWB, BLC...... Image enhancement blcoks

Sensor Driver Requirement

The sensor driver should implement controls in the following table. 

Controls Description R/W Needed by
V4L2_CID_VBLANK Vertical blanking. The idle period after every frame during which no image data is produced. The unit of vertical blanking is a line. Every line has length of the image width plus horizontal blanking at the pixel rate defined by V4L2_CID_PIXEL_RATE control in the same sub-device. R Ae
V4L2_CID_HBLANK Horizontal blanking. The idle period after every line of image data during which no image data is produced. The unit of horizontal blanking is pixels. R Ae
V4L2_CID_EXPOSURE Determines the exposure time of the camera sensor. The exposure time is limited by the frame interval. Drivers should interpret the values as 100 µs units, where the value 1 stands for 1/10000th of a second, 10000 for 1 second and 100000 for 10 seconds. R/W Ae
V4L2_CID_ANALOGUE_GAIN Analogue gain is gain affecting all colour components in the pixel matrix. The gain operation is performed in the analogue domain before A/D conversion. R/W Ae
V4L2_CID_DIGITAL_GAIN Digital gain is the value by which all colour components are multiplied by. Typically the digital gain applied is the control value divided by e.g. 0x100, meaning that to get no digital gain the control value needs to be 0x100. The no-gain configuration is also typically the default. R/W Ae
V4L2_CID_PIXEL_RATE Pixel rate in the source pads of the subdev. This control is read-only and its unit is pixels / second. R Ae
V4L2_CID_LINK_FREQ Data bus frequency. Together with the media bus pixel code, bus type (clock cycles per sample), the data bus frequency defines the pixel rate (V4L2_CID_PIXEL_RATE) in the pixel array (or possibly elsewhere, if the device is not an image sensor). The frame rate can be calculated from the pixel clock, image width and height and horizontal and vertical blanking. While the pixel rate control may be defined elsewhere than in the subdev containing the pixel array, the frame rate cannot be obtained from that information. This is because only on the pixel array it can be assumed that the vertical and horizontal blanking information is exact: no other blanking is allowed in the pixel array. The selection of frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz. R mipi-dphy

Devicetree Bindings

https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-isp1.txt

https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt

Linux

User applications 

The applcation for plain v4L2 capture device still can be used(like gstreamer v4l2src plugins, v4l2ctl), but the media controller must be setup in a proper state before starting capture.
The applications for Video Capture with rockchip-isp1 driver should do things like the following commands:

# Media graph have been initializated when kernel driver probing, so below commands don't have to be called, though 
# please note that they can be changed during runtime.
media-ctl -l '"rockchip-sy-mipi-dphy":1->"rkisp1-isp-subdev":0[1]'
media-ctl -l '"rkisp1-input-params":0->"rkisp1-isp-subdev":1[1]'
media-ctl -l '"rkisp1-isp-subdev":2->"rkisp1_selfpath":0[1]'
media-ctl -l '"rkisp1-isp-subdev":2->"rkisp1_mainpath":0[1]'
media-ctl -l '"rkisp1-isp-subdev":3->"rkisp1-statistics":0[1]'

# Set pad formats
## rkisp1 source pad : format\selection
media-ctl -set-v4l2 '"rkisp1-isp-subdev":2[fmt:YUYV2X8/800x600]'
media-ctl -set-v4l2 '"rkisp1-isp-subdev":2[fmt:YUYV2X8/800x600]' --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/800x600]'

## rkisp1 sink pad : format\selection
media-ctl --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SRGGB10/800*600]'
media-ctl --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SBGGR10/800x600]' --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/800x600]'

## sensor source pad format
media-ctl --set-v4l2 '"ov2685 7-0010":0[fmt:SRGGB10/800*600]'

# Set link(Assume there are two sensors in one mipi-phy)
## Enable sensor1
media-ctl -l '"ov2685 7-0010":0->"rockchip-sy-mipi-dphy":0[1]'
## Disable sensor2
media-ctl -l '"ov5695 7-0036":0->"rockchip-sy-mipi-dphy":0[0]'
# start capture
v4l2-ctl xxxx
or
gst-launch rkcamsrc xxxx

3A

https://github.com/rockchip-linux/gstreamer-rockchip-extra/tree/master/gst/rkv4l2/rkcamsrc/rkisp1

Gstreamer

We have writen a gstreamer plugin, rkcamsrc, which is simply modified from gstreamer v4l2src plugin.

RTENOTITLE

Please refer to readme in github for usage.

Android

Camera Hal3

https://source.android.com/devices/camera/camera3

Reproduce

The driver currently can run on ASUS Tinker Board and Firefly-RK3288-Reload + Rockchip Debian/Yocto SDK.

Source Code

Kernel

  1. 4.15 + FORMLIST Patches
  2. Rockchip 4.4 BSP

Sensor Driver

  1. IMX219
  2. OV5647
  3. TC35874X

Device Tree

  1. Tinker Board
  2. Firefly-rk3288-Reload

 

Test Command

cat /sys/class/video4linux/video*/name
# get self_path videox and main_path videox+1
gst-launch-1.0 rkcamsrc device=/dev/videox+1 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,width=640,height=480 ! kmssink

Others

Topology

Media device information
------------------------
driver          rkisp1
model           rkisp1
serial          
bus info        
hw revision     0x0
driver version  0.0.0

Device topology
- entity 1: rkisp1-isp-subdev (4 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:SBGGR10_1X10/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                <- "rockchip-sy-mipi-dphy":1 [ENABLED]
        pad1: Sink
                <- "rkisp1-input-params":0 [ENABLED]
        pad2: Source
                [fmt:SBGGR10_1X10/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                -> "rkisp1_selfpath":0 [ENABLED]
                -> "rkisp1_mainpath":0 [ENABLED]
        pad3: Source
                -> "rkisp1-statistics":0 [ENABLED]

- entity 2: rkisp1_selfpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 3: rkisp1_mainpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video1
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 4: rkisp1-statistics (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video2
        pad0: Sink
                <- "rkisp1-isp-subdev":3 [ENABLED]

- entity 5: rkisp1-input-params (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video3
        pad0: Source
                -> "rkisp1-isp-subdev":1 [ENABLED]

- entity 6: rockchip-sy-mipi-dphy (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                <- "imx219 2-0010":0 [ENABLED]
        pad1: Source
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                -> "rkisp1-isp-subdev":0 [ENABLED]

- entity 7: imx219 2-0010 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                -> "rockchip-sy-mipi-dphy":0 [ENABLED]