Difference between revisions of "Rockchip-isp1"

From Rockchip open source Document
Jump to: navigation, search
 
(43 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
 
−
NOTE: Under Construction
 
  
 
 
 
 
Line 9: Line 7:
  
 
= Hardware =
 
= Hardware =
 +
 +
More detailed information could be found in TRM chapter "Image Signal Processing", "MIPI D-PHY" , but they are only available under NDA.
  
 
== ISP Details ==
 
== ISP Details ==
 +
 +
ISP comprises with:
 +
 +
*MIPI serial camera interface
 +
*Image Signal Processing
 +
*Many Image Enhancement Blocks
 +
*Crop
 +
*Resize
  
 
=== Block diagram ===
 
=== Block diagram ===
  
−
The datasheet about ISP is currently only available under NDA, so we can't paste the completed block diagram from datasheet to here.
+
The completed block diagram can not be pasted from datasheet to here, below diagram is an abstract version:
 +
 
 +
[[File:Block.png|1200px|Blocks.png]]
  
−
Below diagram is a abstract version:
+
 
  
 
== MIPI Details ==
 
== MIPI Details ==
 +
 +
There are three D-PHY instances in rockchip SoC, their connection are shown as following figure:
 +
 +
[[File:Phy.png|600px|Phy.png]]
  
 
= Software =
 
= Software =
Line 24: Line 38:
 
== Driver ==
 
== Driver ==
  
−
rockchip-isp1 is a V4L2 based driver for Image Signal Processing block on Rockchip SoC. Compared to the earlier driver rk-isp10, it use Media Controller framework, which it's different from plain v4L2. While the plain v4L2 had a view of the device as a plain DMA based image drabber which connects the input data to the host memory, the Media Controller takes into consideration the fact that a typical video device might consist of multiple sub-devices 
+
rockchip-isp1 is a V4L2 based driver for Image Signal Processing block on Rockchip SoC. Compared to the earlier driver rk-isp10, it use Media Controller framework, which it's different from plain v4L2. While the plain v4L2 had a view of the device as a plain DMA based image drabber which connects the input data to the host memory, the Media Controller takes into consideration the fact that a typical video device might consist of multiple sub-devices.
  
 
=== Media Controller Basics ===
 
=== Media Controller Basics ===
Line 30: Line 44:
 
Please read below link carefully, especially if you don't have use it before.
 
Please read below link carefully, especially if you don't have use it before.
  
−
[https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.html https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.htmlIntegrate 3A algorithm(TO BE DONE)]
+
[https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.html https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.html]
  
−
== Linux ==
+
=== Block diagram ===
 +
 
 +
==== File view ====
 +
 
 +
[[File:Code.png|1200px|Code.png]]
 +
 
 +
==== V4l2 view ====
 +
 
 +
[[File:Rockchip-isp1-v4l2.png|800px|Rockchip-isp1-v4l2.png]]
 +
 
 +
{| border="1" class="table_style" style="width: 768px;"
 +
|-
 +
! style="width: 165px;" | Name
 +
! style="width: 131px;" | Type
 +
! style="width: 450px;" | Description
 +
|-
 +
| style="width: 165px;" | rkisp1_mainpath
 +
| style="width: 131px;" | v4l2_vdev, capture
 +
| style="width: 450px;" |
 +
Format: YUV, RAW Bayer; Max resolution: 4416*3312; Support: Crop
 +
 
 +
|-
 +
| style="width: 165px;" | rkisp1_selfpath
 +
| style="width: 131px;" | v4l2_vdev, capture
 +
| style="width: 450px;" |
 +
Format: YUV, RGB; Max resolution: 1920*1080; Support: Crop
 +
 
 +
|-
 +
| style="width: 165px;" | rkisp1-isp-subdev
 +
| style="width: 131px;" | v4l2_subdev
 +
| style="width: 450px;" |
 +
Internal isp blocks; Support: source/sink pad crop.
 +
 
 +
The format on sink pad should be equal to sensor input format, the size should be equal/less than sensor input size.
 +
 
 +
The format on source pad should be equal to vdev output format if output format is raw bayer, otherwise it should be  YUYV2X8. The size should be equal/less than sink pad size.
 +
 
 +
|-
 +
| style="width: 165px;" | rockchip-sy-mipi-dphy
 +
| style="width: 131px;" | v4l2_subdev
 +
| style="width: 450px;" |
 +
MIPI-DPHY Configure
 +
 
 +
|-
 +
| style="width: 165px;" | rkisp1-statistics
 +
| style="width: 131px;" | v4l2_vdev, capture
 +
| style="width: 450px;" |
 +
Provide Image color Statistics information.
 +
 
 +
|-
 +
| style="width: 165px;" | rkisp1-input-params
 +
| style="width: 131px;" | v4l2_vdev, output
 +
| style="width: 450px;" |
 +
Accept params for AWB, BLC...... Image enhancement blcoks
 +
 
 +
|}
  
−
=== User applications  ===
+
=== Sensor Driver Requirement ===
  
−
The applcation for&nbsp;plain v4L2 capture device still can be used(like gstreamer v4l2src plugins, v4l2ctl), but the media&nbsp;controller must be setup in a&nbsp;proper state before starting capture.<br/> The applications for Video Capture with rockchip-isp1 driver&nbsp;should do things like the following commands:
+
The sensor driver should implement controls in the following table.&nbsp;
−
<pre>media-ctl -l '"rockchip-sy-mipi-dphy":1->"rkisp1-isp-subdev":0[1]'
 
−
media-ctl -l '"rkisp1-input-params":0->"rkisp1-isp-subdev":1[1]'
 
−
media-ctl -l '"rkisp1-isp-subdev":2->"rkisp1_selfpath":0[1]'
 
−
media-ctl -l '"rkisp1-isp-subdev":2->"rkisp1_mainpath":0[1]'
 
−
media-ctl -l '"rkisp1-isp-subdev":3->"rkisp1-statistics":0[1]'
 
−
# Above parts of media pipeline have been initializated when kernel driver probing, but please note that they can be changed in runtime
 
−
# Set pad formats
 
−
## rkisp1 source pad
 
−
media-ctl -set-v4l2 '"rkisp1-isp-subdev":2[fmt:YUYV2X8/800x600]'
 
−
## rkisp1 sink pad == sensor source [ad]
 
−
media-ctl --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SRGGB10/800*600]'
 
−
media-ctl --set-v4l2 '"ov2685 7-0010":0[fmt:SRGGB10/800*600]'
 
−
# Set link(Assume there are two sensors in one mipi-phy)
 
−
## Enable sensor1
 
−
media-ctl -l '"ov2685 7-0010":0->"rockchip-sy-mipi-dphy":0[1]'
 
−
## Disable sensor2
 
−
media-ctl -l '"ov5695 7-0036":0->"rockchip-sy-mipi-dphy":0[0]'</pre>
 
  
−
==== Integrate 3A&nbsp;algorithm(TO BE DONE) ====
+
{| border="1" class="table_style"
 +
|-
 +
! Controls
 +
! Description
 +
! R/W
 +
! Needed by
 +
|-
 +
| V4L2_CID_VBLANK
 +
| <span style="font-size:smaller;">Vertical blanking. The idle period after every frame during which no image data is produced. The unit of vertical blanking is a line. Every line has length of the image width plus horizontal blanking at the pixel rate defined by&nbsp;<code>V4L2_CID_PIXEL_RATE</code>&nbsp;control in the same sub-device.</span>
 +
| R
 +
| Ae
 +
|-
 +
| V4L2_CID_HBLANK
 +
| <span style="font-size:smaller;">Horizontal blanking. The idle period after every line of image data during which no image data is produced. The unit of horizontal blanking is pixels.</span>
 +
| R
 +
| Ae
 +
|-
 +
| V4L2_CID_EXPOSURE
 +
| <span style="font-size:smaller;">Determines the exposure time of the camera sensor. The exposure time is limited by the frame interval. Drivers should interpret the values as 100 µs units, where the value 1 stands for 1/10000th of a second, 10000 for 1 second and 100000 for 10 seconds.</span>
 +
| R/W
 +
| Ae
 +
|-
 +
| V4L2_CID_ANALOGUE_GAIN
 +
| <span style="font-size:smaller;">Analogue gain is gain affecting all colour components in the pixel matrix. The gain operation is performed in the analogue domain before A/D conversion.</span>
 +
| R/W
 +
| Ae
 +
|-
 +
| V4L2_CID_DIGITAL_GAIN
 +
| <span style="font-size:smaller;">Digital gain is the value by which all colour components are multiplied by. Typically the digital gain applied is the control value divided by e.g. 0x100, meaning that to get no digital gain the control value needs to be 0x100. The no-gain configuration is also typically the default.</span>
 +
| R/W
 +
| Ae
 +
|-
 +
| V4L2_CID_PIXEL_RATE
 +
| <span style="font-size:smaller;">Pixel rate in the source pads of the subdev. This control is read-only and its unit is pixels / second.</span>
 +
| R
 +
| Ae
 +
|-
 +
| V4L2_CID_LINK_FREQ
 +
| <span style="font-size:smaller;">Data bus frequency. Together with the media bus pixel code, bus type (clock cycles per sample), the data bus frequency defines the pixel rate (<code>V4L2_CID_PIXEL_RATE</code>) in the pixel array (or possibly elsewhere, if the device is not an image sensor). The frame rate can be calculated from the pixel clock, image width and height and horizontal and vertical blanking. While the pixel rate control may be defined elsewhere than in the subdev containing the pixel array, the frame rate cannot be obtained from that information. This is because only on the pixel array it can be assumed that the vertical and horizontal blanking information is exact: no other blanking is allowed in the pixel array. The selection of frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz.</span>
 +
| R
 +
| mipi-dphy
 +
|}
  
−
==== Multi sensors(TO BE DONE) ====
+
=== Devicetree Bindings ===
  
−
==== Multi paths (TO BE DONE) ====
+
[https://github.com/wzyy2/linux/blob/rkisp1/Documentation/devicetree/bindings/media/rockchip-isp1.txt https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-isp1.txt]
  
−
Record, Preview, Still Capture&nbsp;in different resolution&nbsp;at a time
+
[https://github.com/wzyy2/linux/blob/rkisp1/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt]
  
−
=== Gstreamer ===
 
  
−
We have write a gstreamer plugin, [https://github.com/rockchip-linux/gstreamer-rockchip-extra rkcamsrc], which is modified from gstreamer v4l2src plugin.
 
  
−
[[File:Rkcamsrc.png|500px|RTENOTITLE]]
+
== Linux ==
  
−
Please refer to readme in github for usage.
+
=== User applications&nbsp; ===
 +
 
 +
The v4l-utils tool and applications&nbsp;<br/> The v4l-utils tool is a V4L2 development kit maintained by Linuxtv[1] . It provides a set&nbsp;<br/> of V4L2 and media framework related tools for configuring V4L2 sub-device properties,&nbsp;<br/> testing V4L2 devices, and providing development libraries such as libv4l2.so and so on.&nbsp;<br/> This chapter mainly introduces two command-line tools in v4l-utils: media-ctl and v4l2-&nbsp;<br/> ctl&nbsp;<br/>  media-ctl, used to view and configure topology&nbsp;<br/>  v4l2-ctl, used to configure v4l2 controls, capture frames, set cif, isp, sensor&nbsp;<br/> parameters&nbsp;<br/> The format code of different versions of v4l-utils will be different, especially mbus-fmt&nbsp;<br/> part. The version used in this document is v4l-utils-1.14.1 integrated in Linux SDK.&nbsp;
  
 
== Android ==
 
== Android ==
  
−
=== Camera Hal3 ===
+
=== <del>Camera Hal3</del> ===
  
 
[https://source.android.com/devices/camera/camera3?hl=zh-cn https://source.android.com/devices/camera/camera3]
 
[https://source.android.com/devices/camera/camera3?hl=zh-cn https://source.android.com/devices/camera/camera3]
Line 79: Line 172:
 
= Reproduce =
 
= Reproduce =
  
−
The driver currently can run&nbsp;on ASUS Tinker Board and Firefly-RK3288-Reload + Rockchip Linux Debian/Yocto SDK.
+
The driver currently can run&nbsp;on ASUS Tinker Board and Firefly-RK3288-Reload + Rockchip&nbsp;Debian/Yocto SDK.
  
 
== Source Code ==
 
== Source Code ==
Line 86: Line 179:
  
 
#4.15 + [https://patchwork.linuxtv.org/project/linux-media/list/?submitter=7231 FORMLIST Patches]  
 
#4.15 + [https://patchwork.linuxtv.org/project/linux-media/list/?submitter=7231 FORMLIST Patches]  
−
#[https://github.com/rockchip-linux/kernel/tree/isp-early-access Rockchip 4.4 BSP]  
+
#[https://github.com/rockchip-linux/kernel/tree/release-4.4 Rockchip 4.4 BSP]  
  
 
=== Sensor Driver ===
 
=== Sensor Driver ===
  
−
#[https://github.com/rockchip-linux/kernel/blob/isp-early-access/drivers/media/i2c/imx219.c IMX219]  
+
#[https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/imx219.c IMX219]  
−
#[https://github.com/rockchip-linux/kernel/blob/isp-early-access/drivers/media/i2c/ov5647.c OV5647]  
+
#[https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/ov5647.c OV5647]  
−
#[https://github.com/rockchip-linux/kernel/blob/isp-early-access/drivers/media/i2c/tc35874x.c TC35874X]  
+
#[https://github.com/rockchip-linux/kernel/blob/release-4.4/drivers/media/i2c/tc35874x.c TC35874X]  
  
 
=== Device Tree ===
 
=== Device Tree ===
  
−
#[https://github.com/rockchip-linux/kernel/blob/isp-early-access/arch/arm/boot/dts/rk3288-miniarm.dts Tinker Board]  
+
#[https://github.com/rockchip-linux/kernel/blob/release-4.4/arch/arm/boot/dts/rk3288-miniarm.dts Tinker Board]  
−
#[https://github.com/rockchip-linux/kernel/blob/isp-early-access/arch/arm/boot/dts/rk3288-firefly.dtsi Firefly-rk3288-Reload]  
+
#[https://github.com/rockchip-linux/kernel/blob/release-4.4/arch/arm/boot/dts/rk3288-firefly-reload-linux.dts Firefly-rk3288-Reload]  
 +
 
 +
&nbsp;
  
 
== Test Command ==
 
== Test Command ==
 
<pre>cat /sys/class/video4linux/video*/name
 
<pre>cat /sys/class/video4linux/video*/name
 
# get self_path videox and main_path videox+1
 
# get self_path videox and main_path videox+1
−
gst-launch-1.0 rkcamsrc device=/dev/videox+1 io-mode=4 disable-3A=true&nbsp;! videoconvert&nbsp;! video/x-raw,format=NV12,width=640,height=480&nbsp;! kmssink</pre>
+
gst-launch-1.0 rkcamsrc device=/dev/videox+1 io-mode=4&nbsp;! videoconvert&nbsp;! video/x-raw,format=NV12,width=640,height=480&nbsp;! kmssink</pre>
  
 
= Others =
 
= Others =

Latest revision as of 09:57, 23 September 2020

 

Overview

The document below provide basical informations about the rockchip-isp1 driver driver and Image Signal Processing block on Rockchip SoC with examples and details.

Hardware

More detailed information could be found in TRM chapter "Image Signal Processing", "MIPI D-PHY" , but they are only available under NDA.

ISP Details

ISP comprises with:

  • MIPI serial camera interface
  • Image Signal Processing
  • Many Image Enhancement Blocks
  • Crop
  • Resize

Block diagram

The completed block diagram can not be pasted from datasheet to here, below diagram is an abstract version:

Blocks.png

 

MIPI Details

There are three D-PHY instances in rockchip SoC, their connection are shown as following figure:

Phy.png

Software

Driver

rockchip-isp1 is a V4L2 based driver for Image Signal Processing block on Rockchip SoC. Compared to the earlier driver rk-isp10, it use Media Controller framework, which it's different from plain v4L2. While the plain v4L2 had a view of the device as a plain DMA based image drabber which connects the input data to the host memory, the Media Controller takes into consideration the fact that a typical video device might consist of multiple sub-devices.

Media Controller Basics

Please read below link carefully, especially if you don't have use it before.

https://linuxtv.org/downloads/v4l-dvb-apis/uapi/mediactl/media-controller-intro.html

Block diagram

File view

Code.png

V4l2 view

Rockchip-isp1-v4l2.png

Name Type Description
rkisp1_mainpath v4l2_vdev, capture

Format: YUV, RAW Bayer; Max resolution: 4416*3312; Support: Crop

rkisp1_selfpath v4l2_vdev, capture

Format: YUV, RGB; Max resolution: 1920*1080; Support: Crop

rkisp1-isp-subdev v4l2_subdev

Internal isp blocks; Support: source/sink pad crop.

The format on sink pad should be equal to sensor input format, the size should be equal/less than sensor input size.

The format on source pad should be equal to vdev output format if output format is raw bayer, otherwise it should be  YUYV2X8. The size should be equal/less than sink pad size.

rockchip-sy-mipi-dphy v4l2_subdev

MIPI-DPHY Configure

rkisp1-statistics v4l2_vdev, capture

Provide Image color Statistics information.

rkisp1-input-params v4l2_vdev, output

Accept params for AWB, BLC...... Image enhancement blcoks

Sensor Driver Requirement

The sensor driver should implement controls in the following table. 

Controls Description R/W Needed by
V4L2_CID_VBLANK Vertical blanking. The idle period after every frame during which no image data is produced. The unit of vertical blanking is a line. Every line has length of the image width plus horizontal blanking at the pixel rate defined by V4L2_CID_PIXEL_RATE control in the same sub-device. R Ae
V4L2_CID_HBLANK Horizontal blanking. The idle period after every line of image data during which no image data is produced. The unit of horizontal blanking is pixels. R Ae
V4L2_CID_EXPOSURE Determines the exposure time of the camera sensor. The exposure time is limited by the frame interval. Drivers should interpret the values as 100 µs units, where the value 1 stands for 1/10000th of a second, 10000 for 1 second and 100000 for 10 seconds. R/W Ae
V4L2_CID_ANALOGUE_GAIN Analogue gain is gain affecting all colour components in the pixel matrix. The gain operation is performed in the analogue domain before A/D conversion. R/W Ae
V4L2_CID_DIGITAL_GAIN Digital gain is the value by which all colour components are multiplied by. Typically the digital gain applied is the control value divided by e.g. 0x100, meaning that to get no digital gain the control value needs to be 0x100. The no-gain configuration is also typically the default. R/W Ae
V4L2_CID_PIXEL_RATE Pixel rate in the source pads of the subdev. This control is read-only and its unit is pixels / second. R Ae
V4L2_CID_LINK_FREQ Data bus frequency. Together with the media bus pixel code, bus type (clock cycles per sample), the data bus frequency defines the pixel rate (V4L2_CID_PIXEL_RATE) in the pixel array (or possibly elsewhere, if the device is not an image sensor). The frame rate can be calculated from the pixel clock, image width and height and horizontal and vertical blanking. While the pixel rate control may be defined elsewhere than in the subdev containing the pixel array, the frame rate cannot be obtained from that information. This is because only on the pixel array it can be assumed that the vertical and horizontal blanking information is exact: no other blanking is allowed in the pixel array. The selection of frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz. R mipi-dphy

Devicetree Bindings

https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-isp1.txt

https://github.com/torvalds/linux/tree/master/Documentation/devicetree/bindings/media/rockchip-mipi-dphy.txt


Linux

User applications 

The v4l-utils tool and applications 
The v4l-utils tool is a V4L2 development kit maintained by Linuxtv[1] . It provides a set 
of V4L2 and media framework related tools for configuring V4L2 sub-device properties, 
testing V4L2 devices, and providing development libraries such as libv4l2.so and so on. 
This chapter mainly introduces two command-line tools in v4l-utils: media-ctl and v4l2- 
ctl 
 media-ctl, used to view and configure topology 
 v4l2-ctl, used to configure v4l2 controls, capture frames, set cif, isp, sensor 
parameters 
The format code of different versions of v4l-utils will be different, especially mbus-fmt 
part. The version used in this document is v4l-utils-1.14.1 integrated in Linux SDK. 

Android

Camera Hal3

https://source.android.com/devices/camera/camera3

Reproduce

The driver currently can run on ASUS Tinker Board and Firefly-RK3288-Reload + Rockchip Debian/Yocto SDK.

Source Code

Kernel

  1. 4.15 + FORMLIST Patches
  2. Rockchip 4.4 BSP

Sensor Driver

  1. IMX219
  2. OV5647
  3. TC35874X

Device Tree

  1. Tinker Board
  2. Firefly-rk3288-Reload

 

Test Command

cat /sys/class/video4linux/video*/name
# get self_path videox and main_path videox+1
gst-launch-1.0 rkcamsrc device=/dev/videox+1 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,width=640,height=480 ! kmssink

Others

Topology

Media device information
------------------------
driver          rkisp1
model           rkisp1
serial          
bus info        
hw revision     0x0
driver version  0.0.0

Device topology
- entity 1: rkisp1-isp-subdev (4 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:SBGGR10_1X10/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                <- "rockchip-sy-mipi-dphy":1 [ENABLED]
        pad1: Sink
                <- "rkisp1-input-params":0 [ENABLED]
        pad2: Source
                [fmt:SBGGR10_1X10/800x600 field:none
                 crop.bounds:(0,0)/800x600
                 crop:(0,0)/800x600]
                -> "rkisp1_selfpath":0 [ENABLED]
                -> "rkisp1_mainpath":0 [ENABLED]
        pad3: Source
                -> "rkisp1-statistics":0 [ENABLED]

- entity 2: rkisp1_selfpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 3: rkisp1_mainpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video1
        pad0: Sink
                <- "rkisp1-isp-subdev":2 [ENABLED]

- entity 4: rkisp1-statistics (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video2
        pad0: Sink
                <- "rkisp1-isp-subdev":3 [ENABLED]

- entity 5: rkisp1-input-params (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video3
        pad0: Source
                -> "rkisp1-isp-subdev":1 [ENABLED]

- entity 6: rockchip-sy-mipi-dphy (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                <- "imx219 2-0010":0 [ENABLED]
        pad1: Source
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                -> "rkisp1-isp-subdev":0 [ENABLED]

- entity 7: imx219 2-0010 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:SRGGB10_1X10/3280x2464 field:none]
                -> "rockchip-sy-mipi-dphy":0 [ENABLED]
Retrieved from "http://opensource.rock-chips.com/index.php?title=Rockchip-isp1&oldid=2218"