The document below provide basical informations about the rockchip-isp1 driver driver and Image Signal Processing block on Rockchip SoC with examples and details.
2、Hardware
More detailed information could be found in TRM chapter “Image Signal Processing”, “MIPI D-PHY” , but they are only available under NDA.
3、ISP Details
ISP comprises with:
MIPI serial camera interface
Image Signal Processing
Many Image Enhancement Blocks
Crop
Resize
4、Block diagram
The completed block diagram can not be pasted from datasheet to here, below diagram is an abstract version:
5、MIPI Details
There are three D-PHY instances in rockchip SoC, their connection are shown as following figure:
6、Software
7、Driver
rockchip-isp1 is a V4L2 based driver for Image Signal Processing block on Rockchip SoC. Compared to the earlier driver rk-isp10, it use Media Controller framework, which it’s different from plain v4L2. While the plain v4L2 had a view of the device as a plain DMA based image drabber which connects the input data to the host memory, the Media Controller takes into consideration the fact that a typical video device might consist of multiple sub-devices.
8、Media Controller Basics
Please read below link carefully, especially if you don’t have use it before.
| Name | Type | Description | | — | — | — | | rkisp1_mainpath | v4l2_vdev, capture | Format: YUV, RAW Bayer; Max resolution: 4416*3312; Support: Crop | | rkisp1_selfpath | v4l2_vdev, capture | Format: YUV, RGB; Max resolution: 1920*1080; Support: Crop | | rkisp1-isp-subdev | v4l2_subdev | Internal isp blocks; Support: source/sink pad crop. The format on sink pad should be equal to sensor input format, the size should be equal/less than sensor input size. The format on source pad should be equal to vdev output format if output format is raw bayer, otherwise it should be YUYV2X8. The size should be equal/less than sink pad size. | | rockchip-sy-mipi-dphy | v4l2_subdev | MIPI-DPHY Configure | | rkisp1-statistics | v4l2_vdev, capture | Provide Image color Statistics information. | | rkisp1-input-params | v4l2_vdev, output | Accept params for AWB, BLC…… Image enhancement blcoks |
12、Sensor Driver Requirement
The sensor driver should implement controls in the following table.
Controls
Description
R/W
Needed by
V4L2_CID_VBLANK
Vertical blanking. The idle period after every frame during which no image data is produced. The unit of vertical blanking is a line. Every line has length of the image width plus horizontal blanking at the pixel rate defined by V4L2_CID_PIXEL_RATE control in the same sub-device.
R
Ae
V4L2_CID_HBLANK
Horizontal blanking. The idle period after every line of image data during which no image data is produced. The unit of horizontal blanking is pixels.
R
Ae
V4L2_CID_EXPOSURE
Determines the exposure time of the camera sensor. The exposure time is limited by the frame interval. Drivers should interpret the values as 100 µs units, where the value 1 stands for 1/10000th of a second, 10000 for 1 second and 100000 for 10 seconds.
R/W
Ae
V4L2_CID_ANALOGUE_GAIN
Analogue gain is gain affecting all colour components in the pixel matrix. The gain operation is performed in the analogue domain before A/D conversion.
R/W
Ae
V4L2_CID_DIGITAL_GAIN
Digital gain is the value by which all colour components are multiplied by. Typically the digital gain applied is the control value divided by e.g. 0x100, meaning that to get no digital gain the control value needs to be 0x100. The no-gain configuration is also typically the default.
R/W
Ae
V4L2_CID_PIXEL_RATE
Pixel rate in the source pads of the subdev. This control is read-only and its unit is pixels / second.
R
Ae
V4L2_CID_LINK_FREQ
Data bus frequency. Together with the media bus pixel code, bus type (clock cycles per sample), the data bus frequency defines the pixel rate (V4L2_CID_PIXEL_RATE) in the pixel array (or possibly elsewhere, if the device is not an image sensor). The frame rate can be calculated from the pixel clock, image width and height and horizontal and vertical blanking. While the pixel rate control may be defined elsewhere than in the subdev containing the pixel array, the frame rate cannot be obtained from that information. This is because only on the pixel array it can be assumed that the vertical and horizontal blanking information is exact: no other blanking is allowed in the pixel array. The selection of frame rate is performed by selecting the desired horizontal and vertical blanking. The unit of this control is Hz.
The v4l-utils tool and applications The v4l-utils tool is a V4L2 development kit maintained by Linuxtv[1] . It provides a set of V4L2 and media framework related tools for configuring V4L2 sub-device properties, testing V4L2 devices, and providing development libraries such as libv4l2.so and so on. This chapter mainly introduces two command-line tools in v4l-utils: media-ctl and v4l2- ctl media-ctl, used to view and configure topology v4l2-ctl, used to configure v4l2 controls, capture frames, set cif, isp, sensor parameters The format code of different versions of v4l-utils will be different, especially mbus-fmt part. The version used in this document is v4l-utils-1.14.1 integrated in Linux SDK.
Format: YUV, RAW Bayer; Max resolution: 4416*3312; Support: Crop
rkisp1_selfpath
v4l2_vdev, capture
Format: YUV, RGB; Max resolution: 1920*1080; Support: Crop
rkisp1-isp-subdev
v4l2_subdev
Internal isp blocks; Support: source/sink pad crop. The format on sink pad should be equal to sensor input format, the size should be equal/less than sensor input size. The format on source pad should be equal to vdev output format if output format is raw bayer, otherwise it should be YUYV2X8. The size should be equal/less than sink pad size.
rockchip-sy-mipi-dphy
v4l2_subdev
MIPI-DPHY Configure
rkisp1-statistics
v4l2_vdev, capture
Provide Image color Statistics information.
rkisp1-input-params
v4l2_vdev, output
Accept params for AWB, BLC…… Image enhancement blcoks
INIT_LIST_HEAD(&v4l2_dev->subdevs); spin_lock_init(&v4l2_dev->lock); mutex_init(&v4l2_dev->ioctl_lock); v4l2_prio_init(&v4l2_dev->prio); kref_init(&v4l2_dev->ref); //上面都是做一些初始化工作， get_device(dev); //下面将当前设备的对象，赋值给v4l2_dev->dev中，这样的话当前的设备就成了v4l2设备，由于当前设备已经注册过了设备文件，所以后续不需要在重新注册设备文件。 v4l2_dev->dev = dev; if (dev == NULL) { /* If dev == NULL, then name must be filled in by the caller */ if (WARN_ON(!v4l2_dev->name[0])) return -EINVAL; return0; }
/* Set name to driver name + device name if it is empty. */ /* 如果v4l2设备的名字为空，则会将当前设备的名字拷贝为v4l2设备中。*/ if (!v4l2_dev->name[0]) snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s %s", dev->driver->name, dev_name(dev)); //下面是非常重要的一步，即将v4l2设备对象，保存到dev设备中，这个dev可以是platform、char、block等设，我们可以使用dev_get_drvdata(dev)获取到v4l2对象。 if (!dev_get_drvdata(dev)) dev_set_drvdata(dev, v4l2_dev); return0; }
F:\Khadas_Edge_Android_Q\kernel\include\media\v4l2-dev.h /** * video_register_device - register video4linux devices * * @vdev: struct video_device to register * @type: type of device to register, as defined by &enum vfl_devnode_type * @nr: which device node number is desired: * (0 == /dev/video0, 1 == /dev/video1, ..., -1 == first free) * * Internally, it calls __video_register_device(). Please see its * documentation for more details. * * .. note:: * if video_register_device fails, the release() callback of * &struct video_device structure is *not* called, so the caller * is responsible for freeing any data. Usually that means that * you video_device_release() should be called on failure. */ staticinlineint __must_check video_register_device(struct video_device *vdev, enum vfl_devnode_type type, int nr) { return __video_register_device(vdev, type, nr, 1, vdev->fops->owner); }
F:\Khadas_Edge_Android_Q\kernel\drivers\media\v4l2-core\v4l2-dev.c int __video_register_device(struct video_device *vdev, enum vfl_devnode_type type, int nr, int warn_if_nr_in_use, struct module *owner) { int i = 0; int ret; int minor_offset = 0; int minor_cnt = VIDEO_NUM_DEVICES; constchar *name_base;
/* A minor value of -1 marks this video device as never having been registered */ vdev->minor = -1;
/* the release callback MUST be present */ if (WARN_ON(!vdev->release)) return -EINVAL; /* the v4l2_dev pointer MUST be present */ if (WARN_ON(!vdev->v4l2_dev)) return -EINVAL;
/* v4l2_fh support */ spin_lock_init(&vdev->fh_lock); INIT_LIST_HEAD(&vdev->fh_list);
/* Part 1: check device type */ switch (type) { case VFL_TYPE_GRABBER: name_base = "video"; break; case VFL_TYPE_VBI: name_base = "vbi"; break; case VFL_TYPE_RADIO: name_base = "radio"; break; case VFL_TYPE_SUBDEV: name_base = "v4l-subdev"; break; case VFL_TYPE_SDR: /* Use device name 'swradio' because 'sdr' was already taken. */ name_base = "swradio"; break; case VFL_TYPE_TOUCH: name_base = "v4l-touch"; break; default: pr_err("%s called with unknown type: %d\n", __func__, type); return -EINVAL; }
vdev->vfl_type = type; vdev->cdev = NULL; if (vdev->dev_parent == NULL) vdev->dev_parent = vdev->v4l2_dev->dev; if (vdev->ctrl_handler == NULL) vdev->ctrl_handler = vdev->v4l2_dev->ctrl_handler; /* If the prio state pointer is NULL, then use the v4l2_device prio state. */ if (vdev->prio == NULL) vdev->prio = &vdev->v4l2_dev->prio;
/* Part 2: find a free minor, device node number and device index. */ #ifdef CONFIG_VIDEO_FIXED_MINOR_RANGES /* Keep the ranges for the first four types for historical * reasons. * Newer devices (not yet in place) should use the range * of 128-191 and just pick the first free minor there * (new style). */ switch (type) { case VFL_TYPE_GRABBER: minor_offset = 0; minor_cnt = 64; break; case VFL_TYPE_RADIO: minor_offset = 64; minor_cnt = 64; break; case VFL_TYPE_VBI: minor_offset = 224; minor_cnt = 32; break; default: minor_offset = 128; minor_cnt = 64; break; } #endif
/* Pick a device node number */ mutex_lock(&videodev_lock); nr = devnode_find(vdev, nr == -1 ? 0 : nr, minor_cnt); if (nr == minor_cnt) nr = devnode_find(vdev, 0, minor_cnt); if (nr == minor_cnt) { pr_err("could not get a free device node number\n"); mutex_unlock(&videodev_lock); return -ENFILE; } #ifdef CONFIG_VIDEO_FIXED_MINOR_RANGES /* 1-on-1 mapping of device node number to minor number */ i = nr;//这里保存下空闲的次设备号 #else /* The device node number and minor numbers are independent, so we just find the first free minor number. */ /* 查找到第一个空闲的minor号,可以理解成是次设备号*/ for (i = 0; i < VIDEO_NUM_DEVICES; i++) if (video_devices[i] == NULL)//所有注册的video_device都会保存到这个数组中 break; if (i == VIDEO_NUM_DEVICES) { mutex_unlock(&videodev_lock); pr_err("could not get a free minor\n"); return -ENFILE; } #endif //minor_offset一般是0，i就是查找到的空闲次设备号，这里总的次设备支持到256个 vdev->minor = i + minor_offset; //这里num会保存到session_id中，唯一表示一个任务。一般情况下nr == minor vdev->num = nr;//将标准位置为已用。
/* Should not happen since we thought this minor was free */ if (WARN_ON(video_devices[vdev->minor])) { mutex_unlock(&videodev_lock); pr_err("video_device not empty!\n"); return -ENFILE; } devnode_set(vdev); /* 下面这个方法字面意思看起来是获取一个index，但是查看源代码会发现 “这里会去查找不是直系的设备空闲号”，就是说只有video_device不为空，而且v4l2 parent 对象相同都会认为是同类，直接跳过相应的index号*/ vdev->index = get_index(vdev); /* 下面要重点了，这里根据次设备号将当前video_device保存到video_device[]数组中*/ video_devices[vdev->minor] = vdev; mutex_unlock(&videodev_lock);
if (vdev->ioctl_ops) determine_valid_ioctls(vdev); /* 分配字符设备文件*/ /* Part 3: Initialize the character device */ vdev->cdev = cdev_alloc(); if (vdev->cdev == NULL) { ret = -ENOMEM; goto cleanup; }//务必留意这个ioctl,后面子设备中的ioctl都是通过这里查找的。 vdev->cdev->ops = &v4l2_fops; vdev->cdev->owner = owner; ret = cdev_add(vdev->cdev, MKDEV(VIDEO_MAJOR, vdev->minor), 1); if (ret < 0) { pr_err("%s: cdev_add failed\n", __func__); kfree(vdev->cdev); vdev->cdev = NULL; goto cleanup; }
/* Part 4: register the device with sysfs */ vdev->dev.class = &video_class;//根目录sys中有对应的属性可操作。 vdev->dev.devt = MKDEV(VIDEO_MAJOR, vdev->minor);//主设备号为81 vdev->dev.parent = vdev->dev_parent;//这里一般是v4l2_device对象 dev_set_name(&vdev->dev, "%s%d", name_base, vdev->num); ret = device_register(&vdev->dev);//注册该video_device到kernel中。 if (ret < 0) { pr_err("%s: device_register failed\n", __func__); goto cleanup; } /* Register the release callback that will be called when the last reference to the device goes away. */ vdev->dev.release = v4l2_device_release;
/* Part 5: Register the entity. */ ret = video_register_media_controller(vdev);
/* Part 6: Activate this minor. The char device can now be used. */ set_bit(V4L2_FL_REGISTERED, &vdev->flags);
return0;
cleanup: mutex_lock(&videodev_lock); if (vdev->cdev) cdev_del(vdev->cdev); video_devices[vdev->minor] = NULL; devnode_clear(vdev); mutex_unlock(&videodev_lock); /* Mark this video device as never having been registered. */ vdev->minor = -1; return ret; }
F:\Khadas_Edge_Android_Q\kernel\drivers\media\media-device.c /** * media_device_register - register a media device * @mdev: The media device * * The caller is responsible for initializing the media device before * registration. The following fields must be set: * * - dev must point to the parent device * - model must be filled with the device model name */ int __must_check __media_device_register(struct media_device *mdev, struct module *owner) { int ret; if (WARN_ON(mdev->dev == NULL || mdev->model[0] == 0)) return -EINVAL;
F:\Khadas_Edge_Android_Q\kernel\drivers\media\media-devnode.c /** * media_devnode_register - register a media device node * @mdev: media device node structure we want to register * * The registration code assigns minor numbers and registers the new device node * with the kernel. An error is returned if no free minor number can be found, * or if the registration of the device node fails. * * Zero is returned on success. * * Note that if the media_devnode_register call fails, the release() callback of * the media_devnode structure is *not* called, so the caller is responsible for * freeing any data. */ int __must_check media_devnode_register(struct media_devnode *mdev, struct module *owner) { int minor; int ret;
/* Part 1: Find a free minor number */ mutex_lock(&media_devnode_lock); //获取可用的次设备编号。 minor = find_next_zero_bit(media_devnode_nums, MEDIA_NUM_DEVICES, 0); if (minor == MEDIA_NUM_DEVICES) { mutex_unlock(&media_devnode_lock); pr_err("could not get a free minor\n"); return -ENFILE; }
编写.probe()函数，并添加 media control 及 v4l2 sub device 初始化代码作为良好的习惯，完成驱动编码后，也需要增加相应的 Documentation，可以参考Documentation/devicetree/bindings/media/i2c/。这样板级 dts 可以根据该文档快速配置。在板级 dts 中，引用 Sensor 驱动，一般需要
配置正确的 clk，io mux
根据原理图设置上电时序所需要的 regulator 及 gpio
增加 port 子节点，与 cif 或者 isp 建立连接
本章以im214为例，简单分析 Sensor 驱动。
1、上电时序
不同 Sensor 对上电时序要求不同，例如 OV Camera。可能很大部分的 OV Camera 对时序要求不严格，只要 mclk，vdd，reset 或 powerdown 状态是对的，就能正确进行 I2C 通讯并输出图片。但还是有小部分 Sensor 对上电要求非常严格。在 Sensor 厂家提供的 DataSheet 中，一般会有上电时序图，只需要按顺序配置即可。
/* Dequeue a filled buffer */ for (i = 0; i < isp_fmt->mplanes; i++) { u32 payload_size = stream->out_fmt.plane_fmt[i].sizeimage; vb2_set_plane_payload( &stream->curr_buf->vb.vb2_buf, i, payload_size); } stream->curr_buf->vb.sequence = atomic_read(&isp_sd->frm_sync_seq) - 1; stream->curr_buf->vb.vb2_buf.timestamp = ns; vb2_buffer_done(&stream->curr_buf->vb.vb2_buf, VB2_BUF_STATE_DONE); stream->curr_buf = NULL; }
if (!interlaced || (stream->curr_buf == stream->next_buf && stream->u.sp.field == RKISP_FIELD_ODD)) { /* Next frame is writing to it * Interlaced: odd field next buffer address */ stream->curr_buf = stream->next_buf; stream->next_buf = NULL;
/* Set up an empty buffer for the next-next frame */ spin_lock_irqsave(&stream->vbq_lock, lock_flags); if (!list_empty(&stream->buf_queue)) { stream->next_buf = list_first_entry(&stream->buf_queue, struct rkisp1_buffer, queue); list_del(&stream->next_buf->queue); } spin_unlock_irqrestore(&stream->vbq_lock, lock_flags); } elseif (stream->u.sp.field_rec == RKISP_FIELD_ODD && stream->u.sp.field == RKISP_FIELD_EVEN) { /* Interlaced: event field next buffer address */ if (stream->next_buf) { stream->next_buf->buff_addr[RKISP1_PLANE_Y] += stream->u.sp.vir_offs; stream->next_buf->buff_addr[RKISP1_PLANE_CB] += stream->u.sp.vir_offs; stream->next_buf->buff_addr[RKISP1_PLANE_CR] += stream->u.sp.vir_offs; } stream->curr_buf = stream->next_buf; }
stream->ops->update_mi(stream);
if (interlaced) stream->u.sp.field_rec = stream->u.sp.field;
return0; }
/* Update buffer info to memory interface, it's called in interrupt */ staticvoidupdate_mi(struct rkisp1_stream *stream) { structrkisp1_dummy_buffer *dummy_buf = &stream->dummy_buf; printk("zjj.android10.kernel.camera || %s %s %d \n", __FUNCTION__, __FILE__, __LINE__); //zhoujinjian
/* The dummy space allocated by dma_alloc_coherent is used, we can * throw data to it if there is no available buffer. */ if (stream->next_buf) { //NV12中,U和V就交错排布的。看到内存中的排布很清楚，先开始都是Y，之后的都是U1V1U2V2的交错式排布。 //https://blog.csdn.net/byhook/article/details/84037338 //以 4 X 4 图片为例子，占用内存为 4 X 4 X 3 / 2 = 24 个字节,Y存储, mWidth:1280, mHeight:960 printk("zjj.android10.kernel.camera RKISP1_PLANE_Y : 0x%08lx \n", (unsignedlong)stream->next_buf->buff_addr[RKISP1_PLANE_Y]); printk("zjj.android10.kernel.camera RKISP1_PLANE_CB: 0x%08lx \n", (unsignedlong)stream->next_buf->buff_addr[RKISP1_PLANE_CB]); printk("zjj.android10.kernel.camera RKISP1_PLANE_CR: 0x%08lx \n", (unsignedlong)stream->next_buf->buff_addr[RKISP1_PLANE_CR]);
