Mpp
Multimedia Overview on Rockchip platform
MPP: Media Process Platform is the video codec parser andHardware Abstraction Layer library for the Rockchip platforms.
Contents
Get source code
You can get mpp source from git.
git clone -b release https://github.com/rockchip-linux/mpp.git
Build
Unix/Linux
If you are using a Debian relation distribution, there is a debian build rulles in the source code. You may check the debian directory for the future information.
You may use the following command to build the deb packages, supporting architecture are: armhf, arm64.
DEB_BUILD_OPTIONS="parallel=4 nocheck" dpkg-buildpackage -a<arch>
Please install the all requirement cross compiling tools and Debian packages tools before.
You may use the following command to build it in the board:
cmake -DRKPLATFORM=ON -DHAVE_DRM=ON && make
Android
Build in Android need android ndk package, usually we use android-ndk-r10d.
you can download ndk from google website and modify ndk path in make-Android.bash. Auto path is
/home/pub/ndk/android-ndk-r10d.
Then, Carry out build/andorid/xx/make-Android.bash. There may be some errors in the first execution
Re-execution make-Android.bash the probelm can be resolved.
Windows
Carry out build/xx-x86_64/build-all.bat and make-solutions.bat
System Diagram
+---------------------------------------+
| |
| OpenMax / libva |
| |
+---------------------------------------+
+-------------------- MPP ----------------------+
| |
| +-------------------------+ +--------+ |
| | | | | |
| | MPI / MPP | | | |
| | buffer queue manage | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-------------------------+ | | |
| | | | | |
| | codec | | OSAL | |
| | decoder / encoder | | | |
| | | | | |
| +-------------------------+ | | |
| | | |
| +-----------+ +-----------+ | | |
| | | | | | | |
| | parser | | HAL | | | |
| | control | | reg_gen | | | |
| | | | | | | |
| +-----------+ +-----------+ +--------| |
| |
+-------------------- MPP ----------------------+
+---------------------------------------+
| |
| kernel |
| RK vcodec_service / v4l2 |
| |
+---------------------------------------+
About Mpp
The mpp is a middleware library for Rockchip SoC's cross platform media process.
The main purpose of mpp is to provide very high performance, high flexibility
and expansibility on multimedia (mainly video and image) process.
The design target of mpp is to connect different Rockchip hardware kernel driver
and different userspace application.
Rockchip has two sets of hardware kernel driver.
The first one is vcodec_service/vpu_service/mpp_service which is a high
performance stateless frame base hardware kernel driver. This driver supports
all available codecs that hardware can provide. This driver is used on Android/
Linux.
The second one is v4l2 driver which is developed for ChromeOS. It currently
supports H.264/H.265/vp8/vp9. This driver is used on ChomeOS/Linux.
Mpp plans to support serval userspace applications including OpenMax, libva.
Mpp Feature
1. Cross Platform
The target OS platform including Android, Linux, ChromeOS and windows. Mpp uses
cmake to compile on different platform.
2. High Performance
Mpp supports sync / async interface to reduce the time blocked in interface. And
mpp internally make hardware and software run parallelly. When hardware is
running sofware will prepare next hardware task at the same time.
3. High Flexibility
mpi (Media Process Interface) is easy to extend by different control function.
The input/output element packet/frame/buffer is easy to extend different
components.
Mpp Components
OSAL (Operation System Abstraction Layer)
This module shutters the differences between different operation systems and
provide basic components including memory, time, thread, log and hardware memory
allocator.
MPI (Media Process Interface) / MPP
This module is on charge of interaction with external user. Mpi layer has two
ways for user. The simple way - User can use put/get packet/frame function set.
The advanced way - User has to config MppTask and use dequeue/enqueue funciton
set to communicate with mpp. MppTask can carry different meta data and complete
complex work.
Codec (encoder / decoder)
This module implements the high efficiency internal work flow. The codec module
provides a general call flow for different video format. Software process will
be separated from hardware specified process. The software will communicate with
hardware with a common task interface which combines the buffer information and
codec specified infomation.
Parser/Controller and hal (Hardware Abstraction Layer)
This layer provides the implement function call of different video format and
different hardware. For decoder parser provide the video stream parse function
and output format related syntax structure to hal. The hal will translate the
syntax structure to register set on different hardware. Current hal supports
vcodec_service kernel driver and plan to support v4l2 driver later.
Mpp Kernel Driver
Rockchip has two sets of hardware kernel driver.
The first one is vcodec_service/vpu_service/mpp_service which is a high
performance stateless frame base hardware kernel driver. This driver supports
all available codecs that hardware can provide. This driver is used on Android/
Linux.
Here is the vcodec_service kernel driver framework diagram.
+-------------+ +-------------+ +-------------+
| client A | | client B | | client C |
+-------------+ +-------------+ +-------------+
userspace
+------------------------------------------------------------------------------+
kernel
+-------------+ +-------------+ +-------------+
| session A | | session B | | session C |
+-------------+ +-------------+ +-------------+
| | | | | |
waiting done waiting done waiting done
| | | | | |
+---+---+ +---+---+ +---+---+ +---+---+ +---+---+
| task3 | | task0 | | task1 | | task0 | | task0 |
+---+---+ +-------+ +-------+ +-------+ +---+---+
| |
+---+---+ +---+---+
| task2 | | task1 |
+-------+ +-------+
+-----------+
| service |
+---------+-----------+---------+
| | |
waiting running done
| | |
+-----+-----+ +-----+-----+ +-----+-----+
| task A2 | | task A1 | | task C0 |
+-----+-----+ +-----------+ +-----+-----+
| |
+-----+-----+ +-----+-----+
| task B1 | | task C1 |
+-----+-----+ +-----+-----+
| |
+-----+-----+ +-----+-----+
| task A3 | | task A0 |
+-----------+ +-----+-----+
|
+-----+-----+
| task B0 |
+-----------+
The principle of this design is to separate user task handling and hardware
resource management and minimize the kernel serial process time between two
hardware process operation.
The driver uses session as communication channel. Each userspace client (client)
will have a kernel session. Client will commit tasks to session. Then hardware
is managed by service (vpu_service/vcodec_service). Service will provide the
ability to process tasks in sessions.
When client commits a task to kernel the task will be set to waiting status and
link to both session waiting list and service waiting list. Then service will
get task from waiting list to running list and run. When hardware finishs a task
the task will be moved to done list and put to both service done list and
session done list. Finally client will get the finished task from session.
Mpp Buffer
Mpp buffer is the warpper of the buffer used by hardware. Hardware usually can
not use the buffer malloc by cpu. Then we design MppBuffer for different memory
allocator on different platform. Currently it is designed for ion buffer on
Android and drm buffer on Linux. Later may support vb2_buffer in v4l2 devices.
In order to manage buffer usage in different user mpp buffer module introduces
MppBufferGroup as bufffer manager. All MppBuffer will connect to its manager.
MppBufferGroup provides allocator service and buffer reuse ability.
Different MppBufferGroup has different allocator. And besides normal malloc/free
function the allocator can also accept buffer from external file descriptor.
MppBufferGroup has two lists, unused buffer list and used buffer list. When
buffer is free buffer will not be released immediately. Buffer will be moved to
unused list for later reuse. There is a good reason for doing so. When video
resolution comes to 4K the buffer size will be above 12M. It will take a long
time to allocate buffer and generate map table for it. So reusing the buffer
will save the allocate/free time.
Here is the diagram of Mpp buffer status transaction.
+----------+ +---------+
| create | | commit |
+-----+----+ +----+----+
| |
| |
| +----v----+
+----------+ unused <-----------+
| +---------+ |
| | | |
| | | |
+-----v----+ | | +-----+----+
| malloc | | | | free |
+----------+ | | +----------+
| inc_ref | +---------+ | dec_ref |
+-----+----+ +-----^----+
| |
| |
| +---------+ |
+----------> used +-----------+
+---------+
| |
| |
| |
| |
| |
+----^----+
|
|
+----+----+
| import |
+---------+
Mpp Task
Mpp task is the contain component for transaction with external user in advanced
mode. The target of advanced mode is to provide flexible, multiple input/output
content for extension.
Mpp task has mpp_meta as the rich content carrier. Mpp meta uses KEY and value
pair for extension. One task can carries multiple data into or out of mpp.
The typical case is encoder with OSD and motion detection. One task may contain
OSD buffer, motion detection buffer, frame buffer and stream buffer as input and
output stream buffer and motion detection buffer with data. And this case can
also be used on decoder if decoder wants to output some side information.
1. Mpp task queue
Mpp task queue is the manager of tasks. Due to user may incorrectly use the task
we choose the design that hold all task inside mpp. Task queue will create and
release task. But task queue will not interact with user directly. We use port
and task status to control the transaction.
2. Mpp port
Mpp port is the transaction interface of task queue. External user and internal
worker thread will use mpp_port_poll / mpp_port_dequeue / mpp_port_enqueue
interface to poll / dequeue / enqueue the task task queue. Mpp advanced mode is
using port to connect external user, interface storage and internal process
thread. Each task queue has two port: input port and output port. And from a
global view the task will always flow from input port to output port.
3. Mpp task status
There are four status for one task. Mpp use list_head to represent the status.
INPUT_PORT : Initial status for input port user to dequeue. Or when output port
successfully enqueue a task then the task is on this status.
INPUT_HOLD : When input port user successfully dequeue a task then the task is
on this status.
OUTPUT_PORT: When input port user successfully enqueue a task then the task is
on OUTPUT_PORT status. And this task is ready for dequeue from
output port.
OUTPUT_HOLD: When output port user successfully dequeue a task then the task is
on this status.
4. Mpp task / port transaction
There are three transaction functions on a port: poll / dequeue / enqueue.
When port user call the transaction function task will be transfer from one
status to next status. The status transform flow is unidirectional from input
port to output port.
The overall relationship graph of task / port / status is shown below.
1. task queue
+------------------------------+
| |
+----+----+ +--------------+ +----+----+
| 4 | | 3 | | 2.1 |
+--------+ dequeue <--+ status <--+ enqueue <---------+
| | | | INPUT_PORT | | | |
| +---------+ | | +---------+ |
+------v-----+ | | +--------------+ | | +------+------+
| 3 | | 2 | | 2 | | 3 |
| status | | input | | output | | status |
| INPUT_HOLD | | port | | port | | OUTPUT_HOLD |
| | | | | | | |
+------+-----+ | | +--------------+ | | +------^------+
| +---------+ | 3 | +---------+ |
| | 4 | | status | | 2.1 | |
+--------> enqueue +--> INPUT_PORT +--> dequeue +---------+
| | | | | |
+----+----+ +--------------+ -----+----+
| |
+------------------------------+
On advanced mode mpp uses two task queue: input task queue and output task
queue.
Input task queue connects input side external user to internal worker thread.
Output task queue connects internal worker thread to output side external user.
Then there will be three threads to parallelize internal process and external
transation. This will maximize mpp efficiency.
The work flow is demonstrated as below graph.
+-------------------+ +-------------------+
| input side user | | output side user |
+----^---------+----+ +----^---------+----+
| | | |
+----+----+----v----+ +----+----+----v----+
| dequeue | enqueue | | dequeue | enqueue |
+----^----+----+----+ +----^----+----+----+
| | | |
+----+---------+----+ MPP +----+---------v----+
+---+ input port +-----------+ output port +---+
| +-------------------+ +-------------------+ |
| | input task queue | | output task queue | |
| +-------------------+ +-------------------+ |
| | output port | | input port | |
| +----+---------^----+ +----+---------^----+ |
| | | | | |
| +----v----+----+----+ +----v----+----+----+ |
| | dequeue | enqueue | | dequeue | enqueue | |
| +----+----+----^----+ +----+----+----^----+ |
| | | | | |
| +----v---------+---------------------v---------+----+ |
| | internal work thread | |
| +---------------------------------------------------+ |
| |
+-----------------------------------------------------------+
Mpp User Guide
Multithreading
Mpp don't directly provide thread api, so you should use the correspoding
os interface.
About multithreading encode/decode, something must be done.
1. Resources should apply for multiple copies rather than one, To avoid the
competition of resources
2. The timing of resource release is different, release resource should be done after
a frmae or a packet end to used.
For specific details about multithreading, see test/mpi_enc_test and test/mpi_dec_test.
param set
Mpp nees some param to decide work mode, param include MPP, OSAL, CODEC, ISP, HAL
and Mode module cmd.
All cmd is define in Inc/Rk_mpi_cmd.h
Usually cmd is set after mpp_create and mpp_init, but work mode should be set
before mpp_init. you need to indicate work mode(single or advanced) to help mpp cho
se thread.
typedef enum {
MPP_OSAL_CMD_BASE = CMD_MODULE_OSAL,
MPP_OSAL_CMD_END,
MPP_CMD_BASE = CMD_MODULE_MPP,
MPP_ENABLE_DEINTERLACE,
MPP_SET_INPUT_BLOCK,
MPP_SET_OUTPUT_BLOCK,
MPP_CMD_END,
MPP_CODEC_CMD_BASE = CMD_MODULE_CODEC,
MPP_CODEC_GET_FRAME_INFO,
MPP_CODEC_CMD_END,
MPP_DEC_CMD_BASE = CMD_MODULE_CODEC | CMD_CTX_ID_DEC,
MPP_DEC_SET_FRAME_INFO, /* vpu api legacy control for buffer slot dimension init */
MPP_DEC_SET_EXT_BUF_GROUP, /* IMPORTANT: set external buffer group to mpp decoder */
MPP_DEC_SET_INFO_CHANGE_READY,
MPP_DEC_SET_INTERNAL_PTS_ENABLE,
MPP_DEC_SET_PARSER_SPLIT_MODE, /* Need to setup before init */
MPP_DEC_SET_PARSER_FAST_MODE, /* Need to setup before init */
MPP_DEC_GET_STREAM_COUNT,
MPP_DEC_GET_VPUMEM_USED_COUNT,
MPP_DEC_SET_VC1_EXTRA_DATA,
MPP_DEC_SET_OUTPUT_FORMAT,
MPP_DEC_CMD_END,
MPP_ENC_CMD_BASE = CMD_MODULE_CODEC | CMD_CTX_ID_ENC,
/* basic encoder setup control */
MPP_ENC_SET_ALL_CFG, /* set MppEncCfgSet structure */
MPP_ENC_GET_ALL_CFG, /* get MppEncCfgSet structure */
MPP_ENC_SET_PREP_CFG, /* set MppEncPrepCfg structure */
MPP_ENC_GET_PREP_CFG, /* get MppEncPrepCfg structure */
MPP_ENC_SET_RC_CFG, /* set MppEncRcCfg structure */
MPP_ENC_GET_RC_CFG, /* get MppEncRcCfg structure */
MPP_ENC_SET_CODEC_CFG, /* set MppEncCodecCfg structure */
MPP_ENC_GET_CODEC_CFG, /* get MppEncCodecCfg structure */
/* runtime encoder setup control */
MPP_ENC_SET_IDR_FRAME, /* next frame will be encoded as intra frame */
MPP_ENC_SET_OSD_PLT_CFG, /* set OSD palette, parameter should be pointer to
MppEncOSDPlt */
MPP_ENC_SET_OSD_DATA_CFG, /* set OSD data with at most 8 regions, parameter
should be pointer to MppEncOSDData */
MPP_ENC_GET_OSD_CFG,
MPP_ENC_SET_EXTRA_INFO,
MPP_ENC_GET_EXTRA_INFO, /* get vps / sps / pps from hal */
MPP_ENC_SET_SEI_CFG, /* SEI: Supplement Enhancemant Information, parameter
is MppSeiMode */
MPP_ENC_GET_SEI_DATA, /* SEI: Supplement Enhancemant Information, parameter
is MppPacket */
MPP_ENC_CMD_END,
MPP_ISP_CMD_BASE = CMD_MODULE_CODEC | CMD_CTX_ID_ISP,
MPP_ISP_CMD_END,
MPP_HAL_CMD_BASE = CMD_MODULE_HAL,
MPP_HAL_CMD_END,
MPI_CMD_BUTT,
} MpiCmd;
you can call like this.
ret = mpi->control(ctx, xx_xx_cmd, &data);
Rate Control
Mpp support three mode of rate control, CBR(constant bitrate),VBR(variable
bitrate) and CQP(FIX QP). Mpp control rate according to the set of three
parameters, include MppEncCodecCfg, MppEncPrepCfg and MppEncRcCfg.
MppEncPrepCfg
+-----------+----------------------------------------+
| param | effect |
| | |
+----------------------------------------------------+
| | |
| format | input pix format, such as NV12, RGB24|
| | |
+----------------------------------------------------+
| | |
| width | wdith of picture |
| | |
+----------------------------------------------------+
| | |
| height | height of picture |
| | |
+----------------------------------------------------+
| | |
| stride | span of a line in picture, must align|
| | in 8 |
+-----------+----------------------------------------+
MppEncRcCfg
+-----------+----------------------------------------+
| param | effect |
| | |
+----------------------------------------------------+
| | |
| rc_mode | 0:CBR |
| | 1:VBR |
+----------------------------------------------------+
| | |
| bps_target| target bitrate |
| | |
+----------------------------------------------------+
| | |
|bps_max/min max/min value of target bitrate |
| | |
+----------------------------------------------------+
| | |
|fps_in_num | input frame rate / denorm |
| /denorm | |
+----------------------------------------------------+
| | |
|fps_out_num output frame rate / denorm |
| /denorm | |
+----------------------------------------------------+
| gop | group of picture, usually set as 4 * |
| | fps |
+-----------+----------------------------------------+
| | |
| uality | profile of VBR mode, worst,worse, |
| | medium,better,best and CQP,note that |
| | it will let qp_max/qp_min lose effect. |
| | |
| | {31, 51}, // worst |
| | {28, 46}, // worse |
| | {24, 42}, // medium |
| | {20, 39}, // better |
| | {16, 35}, // best |
| | {0, 0}, // cqp |
| | cqp means use qp_init,qp will be const |
| | in each frame. |
+-----------+----------------------------------------+
MppEncCodecCfg
+-----------+----------------------------------------+
| param | effect |
| | |
+----------------------------------------------------+
| | |
| qp_init | usually set as 33, if set at 0, mpp |
| | will estimate it by width,height... |
+----------------------------------------------------+
| | |
| qp_min/qp_| min/max value of qp |
| max | |
+----------------------------------------------------+
| | |
| qp_max_step The maximum range of QP differences |
| | between two frames |
+----------------------------------------------------+
| profile | 44 CAVLC 4:4:4 |
| | 66 Baseline |
| | 77 Main |
| | 88 Extended |
| | 100 High (suggest) |
+----------------------------------------------------+
| level | 10 Level 1.0 |
| | 99 Level 1.b |
| | 11 Level 1.1 |
| | 12 Level 1.2 |
| | 13 Level 1.3 |
| | 20 Level 2.0 |
| | 21 Level 2.1 |
| | 22 Level 2.2 |
| | 30 Level 3.0 |
| | 31 Level 3.1 |
| | 32 Level 3.2 |
| | 40 Level 4.0 |
| | 41 Level 4.1 |
| | 42 Level 4.2 |
| | 50 Level 5.0 |
| | 51 Level 5.1 |
+-----------+----------------------------------------+
| entroy_cod| entroy code mode 1:CAVLC |
| ing_mode | 0:CABAC |
+----------------------------------------------------+
| cabac_init| effect when entroy mode is CABAC |
| _idc | |
+-----------+----------------------------------------+
1. CBR
In CBR mode, the bitrate of encoded data is controlled whtnin the configured maximum
bit rate and the minimum bitrate range.
suggest param +-----------+----------------------------------------+ | param | effect | | | | +----------------------------------------------------+ | | | | rc_mode | MPP_ENC_RC_MODE_CBR | +----------------------------------------------------+ | qp_init | 0(suggest, mpp internal will set it) | | | 24(high bitrate) | | | 32(medium bitrate) | | | 40(low bitrate) | +----------------------------------------------------+ | qp_min | suggest 0, mpp internal will set at 16 | | | | +----------------------------------------------------+ | qp_max | suggest 0, mpp internal will set at 48 | | | | +-----------+----------------------------------------+ | qp_max_step 16 | +----------------------------------------------------+
2. VBR
In VBR mode, mpp will try to keep the video quality stable and make the code rate of
the encoded file fluctuate as much as possible within the given maximum and minimum
bit rate ranges.
suggest param +-----------+----------------------------------------+ | param | effect | | | | +----------------------------------------------------+ | rc_mode | MPP_ENC_RC_MODE_VBR | +----------------------------------------------------+ | quality | worst(very low bitrate) | | | worse(low bitrate) | | | medium(medium bitrate) | | | better(high bitrate) | | | best(very high bitrate) | | | CQP(const qp,rate control lose effect| +----------------------------------------------------+ | qp_init | 0(suggest, mpp internal estimate) | | | 24(high bitrate) | | | 32(medium bitrate) | | | 40(low bitrate) | +----------------------------------------------------+ | qp_min | 0 | +----------------------------------------------------+ | qp_max | 0 | +-----------+----------------------------------------+ | qp_max_step 8, set at 0 when CQP | +----------------------------------------------------+
Mpp Interface Use
1. MPP_RET Mpp::put_packet(MppPacket packet)
Is packaged as mpi->put_packet, and for interact with internal packet list in decoder.
2. MPP_RET Mpp::get_frame(MppFrame *frame)
Is packaged as mpi->get_frame, and for interact with internal frame list in decoder. you can
call like this when decode.
mpi->put_packet
|
mpi->get_frame
3. MPP_RET Mpp::put_frame(MppFrame frame)
Is packaged as mpi->put_frame, and for interact with internal frame list in encoder.
4. MPP_RET Mpp::get_packet(MppPacket *packet)
Is packaged as mpi->get_packet, and for interact eith internal packet list in encoder. you can
call like this when encode.
mpi->put_frame
|
mpi->get_packet
5. MPP_RET Mpp::poll(MppPortType type, MppPollType timeout)
Is packaged as mpi->poll, poll usually used in task mode(advance mode). poll will check internal
task list, if MPP_INPUT_PORT/MPP_OUTPUT_PORT is empty, poll will block util task list get node,
else poll will return immediately.
about how does task modework, please see doc/design/4.mpp_task.txt
6. MPP_RET Mpp::dequeue(MppPortType type, MppTask *task)
Is packaged as mpi->dequeue, will get a task with multiple data. when port is MPP_INPUT_PORT, get a
task to set. when port is MPP_OUTPUT_PORT, get a task with encoded data.
7. MPP_RET Mpp::enqueue(MppPortType type, MppTask task)
Is packaged as mpi->enqueue, will send a task to internal task list, so that list get a node. you
can call like this when encode.
mpi->poll(input, xx)
|
mpi->dequeue(input, xx)
|
mpi->enqueue(input, xx)
|
mpi->poll(output, xx)
|
mpi->dequeue(output, xx)
|
mpi->enqueue(output, xx)
8. MPP_RET Mpp::control(MpiCmd cmd, MppParam param)
Is packaged as mpi->control, will change mpp param.
About how to use mpp encode/decode, please test/mpi_encode_test.c and test/mpi_decode_test.c
Mpp Development Reference Manual
http://opensource.rock-chips.com/wiki_File:MPP_Development_Reference.pdf
Mpp Development Guide (CN)
http://opensource.rock-chips.com/wiki_File:MPP_%E5%BC%80%E5%8F%91%E5%8F%82%E8%80%83_v0.3.pdf
