Sony is not only pushing the frame rate boundaries of CMOS sensors with their Stacked technology; but now it’s implementing and improving on the design with circuitry that can track objects with pinpoint electronic precision based on pixel colors, contrast, and motion vectors. The result the IMX382 is a complete tracking sensor solution that could one day be used for full AF during high-speed video capture by accurately representing the scene.
While 1000fps is not the end all be all of high-speed frame rate, the fact that it can accurately track with very little lag the object motion in front could one day translate into perfect autofocus capability for many types of cameras. Not only the focus department is improved by this technology but now you can use the sensor in all sorts of industrial, law enforcement, traffic cameras and monitoring solutions for robotics.
Sony New IMX382 Sensor Could Change Everything:
The demo shows the technology on things like coin counting machines that have to scan at superhuman speeds. But we can glimpse at a future where the camera is intelligent enough to track you, focus and become a full camera assistant that pulls focus precisely, becoming the ultimate Vlogger tool that will not break the bank or require a salary.
Auto Focus systems today do show promise by leveraging the power of phase detection along with intelligent algorithms to keep subjects in focus. Canon has as of now the most natural looking and dependable solution with its dual pixel AF technology present in cameras like the 80D and the 5D Mark IV. However these systems are not intelligently tracking subjects based on the preference on the shooter, with a few machine learning algorithms it could use the new Sensor’s 1000fps power to create a customizable tracking solution that could communicate motion to a free motion base or drone that tracks subjects and focuses accordingly without the ill effects of misfocus.
The sensing power of the sensor is limited to just above HD resolution at 500fps with 4 bits or 1000fps at just under VGA. This is not a problem for image quality as this is used only by sensing; not image capture.
| 500fps 1000fps |
Quad-VGA (1280×960) 4bit 640×470 4bit |
Sony is making leaps in technology with their sensors that are leapfrogging the rest of the market. In a few years, the Stacked sensor technology will probably be present in most cameras, phones, and monitoring solutions. Robotics are in need of this technology as well by needing hyper fast optical input for decision making.
Will Autonomous cars use the new sensor?
Level 3 autonomy in cars will require the best sensors possible and ones that can detect in fractions of a second if the obstacle in front is a person, a deadly object or something soft. Cars will be making decisions that will either save lives or take them based on the least damage possible. Sensors like these could scan the scene much faster and be able to analyze things like motion of target, colors, and shapes before feeding it to the main onboard computers and make car autonomy safer than any test before it. The future will be driverless cars and technology like this will make it possible.
How Google self-driving car sees a road by Yury Yurevich:
https://www.youtube.com/watch?v=MqUbdd7ae54
We hope to see these sensors in cameras soon, but take a look at the test cases below withing the press release by Sony. It is quite an impressive technology. High speed is no longer just for shooting invisible time stamps but now could be incorporated in devices all over the environment.
You will now be monitored at 1000fps, so even magicians will have to up their game when playing card tricks!
Full Press Release and Demo Video Below:
Sony Releases a High-Speed Vision Sensor that Makes Detection and Tracking of Objects at 1,000 fps Possible
Stacked CMOS Image Sensor that Combines High-Frame-Rate Imaging and High-Speed Sensing
Sony Semiconductor Solutions Corporation
Tokyo, Japan—Sony Corporation today announced the release of the new IMX382 high-speed vision sensor, which enables detection and tracking of objects at 1,000 frames per second. Sony will begin shipping samples in October 2017.
This vision sensor features a stacked configuration with a back-illuminated pixel array and signal processing circuit layer. The circuit layer is equipped with image processing circuits and a programmable column-parallel processor, delivering high-speed target detection and tracking. This configuration enables high-sensitivity imaging thanks to the back-illuminated pixel array, with target detection and tracking at 1,000 frames per second. The new sensor uses information such as color and brightness obtained from pixels to detect objects, then extracts the object’s centroid, moment and motion vector, and finally outputs the information from the vision sensor in each frame.
High-speed vision sensor IMX382
| Model name | Sample shipment date | Sample price (excluding tax) |
|---|---|---|
| High-speed vision sensor IMX382 | October 2017 | 100,000 JPY |
This vision sensor has achieved imaging, target detection and tracking at 1,000 fps on a single chip. This not only expands the scope of application but delivers new value that was difficult to obtain with conventional systems.
On past systems equipped with a 30 fps processing image sensor, it was not always possible to capture fast-moving objects and/or phenomena. At 1,000 fps, approximately 33 times faster in imaging speed than conventional chips, the new product can capture fast-moving objects and detect objects from image information, while using high-speed processing to extract information such as centroid, moment, and motion vector. Because the processing results can be output from the sensor in each frame, feedback to the system is faster than conventional methods. Delays in detecting anomalies or malfunctions in production lines at factories and other places can produce fatal results. However, this vision sensor makes it possible to instantly capture such events, so that orders can be issued quickly to stop the system.
Conventional industrial robots are normally operated by specifying motion coordinates using a program. Utilizing the high-speed tracking functionality of this vision sensor, however, enables real-time feedback to robots, enabling autonomous robot operation that is responsive to the movement and status of objects. This helps make robot teaching more efficient, contributing to improved productivity.
On conventional systems, the image processing required for detecting and tracking of objects has to be carried out at a later stage (on a computer, or other computing device). This vision sensor changes all that by delivering imaging, target detection and tracking image processing on a single chip. Not only does this make the later stage devices more compact and help to save overall system power; it also expands the new system development potential by eliminating certain physical limitations.
Timed with the sample shipping, Sony will also be providing evaluation kits. The evaluation kit consists of the camera and control software, making it easy to evaluate the sensor in diverse customer environments.
Sony is focusing intently on the sensing field and seeking to leverage its potential in the image sensor market. By integrating the imaging technology that it has developed over the years with sensing technologies that enable information acquisition and utilization, Sony strives to pioneer new applications and new markets for image sensors.
Main Features
- 1. Enables detection and tracking of targets at 1,000 frame per second
-
- The signal processing circuit layer on the bottom of the stacked configuration is equipped with image processing circuits and a programmable column-parallel processor, delivering high-speed target detection and tracking. This construction enables high-sensitivity imaging with back-illuminated pixel array and achieves target detection and tracking at 1,000 frames per second.
- The new sensor uses information such as color and brightness obtained from pixels to detect objects, then extracts the object’s centroid, moment and motion vector, and finally outputs the information from the vision sensor as a signal in each frame.
- 2. Built-in sensing function for a compact system and power savings
-
- The product achieves imaging and image processing on a single chip. Not only does this make for a more compact overall system and power savings; it also expands the new system development potential by eliminating certain physical limitations.
- For system output, in addition to the image data output pathway the chip is also equipped with an output pathway solely for sensing information, so users can select between them for their particular application.
- 3. Evaluation kit allows the sensor to be evaluated in diverse usage environments
- The evaluation kit consists of a camera and software and is designed makes it easy to evaluate the sensor in a variety of customer usage environments.
IMX382 high-speed vision sensor configuration
Key Specifications
| Number of effective pixels | 1304 (H) ×976 (V), 1.27 megapixels | ||
|---|---|---|---|
| Image size | Diagonal 5.68 mm (type 1/3.2) | ||
| Unit cell size | 3.5μm (H) × 3.5μm (V) | ||
| Frame rate | Viewing | 60fps 120fps |
Quad-VGA (1280×960) 12bit 640×470 12bit |
| Sensing | 500fps 1000fps |
Quad-VGA (1280×960) 4bit 640×470 4bit |
|
| Power supply | 2.9V / 1.8V / 3.3V / 1.1V | ||
| Image format | Bayer RAW / Black and white | ||
| Output | MIPI (CSI2) D-PHY 864Mbps/lane | ||
More information on this new sensor at the Sony site here:
https://www.sony.net/SonyInfo/News/Press/201705/17-051E/index.html