Yes, the Nokia 808 PureView has the largest-ever sensor by a long way shoe-horned into its pocketable dimensions. When people hear the figures, many either find their jaws on the floor in sheer astonishment or struggle to believe it’s possible. After all, this isn’t a digital SLR (that would be astonishing enough) but a smartphone! Something you can carry with you at all times.
I can understand the reactions: even people inside Nokia have reacted similarly.
Despite this, the innovation and news is NOT the number of pixels but rather HOW those pixels are used.
It’s been incredibly exciting to have been associated with this project from a very early stage. For some of our team, it’s taken over five years to bring this to the market, such is the technological and engineering achievement, so you can perhaps imagine the excitement but also sense of relief some of us are feeling right now.
Given the amount of effort that’s gone into this project, I wanted to share more of the background as well as some more detail around how PureView works.
Where it all started
In late 2005, Nokia were in the final phases of preparing the Nokia N73 3Mpix AF and the rather unique N93 3Mpix AF 3x optical zoom smartphones for introduction in the spring of 2006. We’d already been researching alternative directions in the area of imaging and camera development as well as extending the direction both of these products would be soon starting. Roughly a year after their introduction, the N95 and N93i came to market.
Around this time, we were starting the development of a number of next-generation imaging rich smartphones. Commercial products such as the Nokia N82, N86 8MP as well as the extremely popular Nokia N8. But there were many other projects intended to include optical zoom which never made it to the market. A number of these were quite advanced concepts using different camera configurations and physical form factors, some conventional, some significantly different.
However, over this time, the market was evolving. For example, displays were becoming bigger and bigger. This aspect alone resulted in a number of concepts not being taken forward due to the limited potential screen size of some concepts. Another important factor was how market expectations were evolving in the area of image quality.
For example, at one stage we had working prototypes equipped with optical zoom using folded optics. Despite this almost reaching commercialization, the module was relatively large and we decided the performance would not be fundamentally good enough to meet the evolving expectations.
It became clear to us that if we were ever to meet the increasing expectations and evolving market dynamics we were going to need to find a new direction in imaging.
After developing several optical zoom modules, we were still seeing significant performance trade-offs caused by optical zoom: performance in low light; image sharpness at both ends of the zoom range; audible noise problems; slow zooming speed and lost focus when zooming during video. We became convinced this could never be the great experience we once hoped. You’d need to accept a bigger, more expensive device with poor f no., a small and noisy image sensor and lower optical resolution just to be able to zoom.
Around this time, the Nokia imaging team had just finished creating a tool called the Camera Simulation Environment. This tool is a virtual environment where we can easily simulate the performance of different types of optics, image sensors and image processing algorithms and see the impact of different technical solutions to the final image quality. It’s an easy and fast way to try new ideas.
Nokia was also leading the market by driving large image sensors into devices and understood how to integrate large image sensors in to small camera modules. The Nokia N73 and N95 were the first mobile products with 1/2.5” sensors and since then we’ve continued to introduce large sensors such as the 1/1.83” sensor in the Nokia N8.
Of course, we understood the need for being able to zoom and frame the shot during video recording. However, compromising image and video quality to achieve the zooming capability was something we were not willing to do.
One idea leads to another
One day when a couple of our engineers met over lunch, one of them mentioned how earlier that day he found an article in the Electronics Times on satellite imaging inspiring, specifically how satellite imaging uses extremely high resolution sensors to capture high resolution images. It was the fact that we typically only ever look at a section of a satellite image that inspired him the most.
An idea emerged from this discussion to use a sensor with somewhat higher resolution sensor than needed at the time but output a lower resolution image than the sensor input resolution possibly adding some upscaling/interpolation to provide a meaningful enough zoom range. This would provide the user with an experience similar to optical zoom. Whilst the performance was thought to be superior to conventional digital zoom as well as result in a far smaller package than optical zoom, it was felt that the performance would still not be up to the standard we were aiming to achieve.
Sometime later after a ten-hour long meeting seeking to solve the technical challenges of optical zooming, a few engineers were sitting in a Tokyo hotel bar. During a lively discussion about how the technical problems of optical zooming could be solved the earlier idea came up again in conversation…. What if we would just add enough pixels to avoid having to upscale the image?
….after some further discussion they concluded that a sufficiently large enough image sensor could create an output image with excellent low light performance, excellent optical performance as well as maintaining a low f no. Instead of trade-offs, there would be significant benefits, especially at the wide range of the zoom. As an additional benefit the file sizes would be small due to low noise whilst the level of detail would be way beyond anything seen before thanks to the pixel oversampling.
At full zoom, while pixel oversampling could not be used, optical performance would benefit as only the central optical path would be used, where the performance is always superior due to manufacturing tolerances and light incoming angle. We could therefore keep the same low f no. and achieve performance which is not possible with optical zooming (not even in expensive SLR optics. As a bonus the closest focus distance would remain the same as wide, resulting in greater macro performance!
We would also achieve instant and silent zooming by keeping the focus during zooming which has always been a problem in optical zooms. We would also be able to achieve simpler, smaller and more robust construction for the camera. Eureka! The solution was right there!
That evening the basic idea had been sketched on a bar napkin, but even during ‘the morning after’ it was clear this idea was really worth taking seriously.
In order to make the camera happen, the largest and highest resolution image sensor in mobile devices would need to be created. Simulations showed that we would need new solutions and materials in the optics to be able to achieve great optical performance in a small enough package. Manufacturing tolerances, materials and surface accuracy used in SLRs, pocket cameras or mobile cameras would not be enough to make it work. Working closely with Carl Zeiss, we analysed different optical solutions, materials and manufacturing technologies, searching the world for image sensor technologies and companies willing to take on the challenge.
We had often debated that, for the vast majority, 5-megapixels completely fulfils their real world needs, but the market for many years has been pixels, pixels, pixels. It’s hard to block that out. Our friends at Carl Zeiss believed the same. At the time, the challenge was like Columbus trying to convince people the world was round and not flat.
Shaping the sensor
At this time, the sensor was supporting the conventional 4:3 aspect ratio. 4:3 aspect ratios were the norm but we could see the future was 16:9. The challenge was how to support 4:3 and 16:9. This part of the story I remember well as I was in the meeting when we brainstormed this part of the module design.
People from Nokia were in the meeting, of course, but also our friends from the companies we work with often on our high-end optics and sensors. The atmosphere was relaxed but I had a feeling that some of our optics and sensor suppliers thought we were perhaps crazy. Nevertheless, they were still putting 100% into the project. We were really pushing the boundaries of optical design at this point clearly going where no one had dared before.
In this meeting we created the idea to use the 13:9 sensor based around the optical circle to fully support both 16:9 and 4:3. Of course, since then we have been incorporating this into the new modules for example in the N9, Lumia 800 and 900. But to maintain the same effective zoom range someone quickly pointed out we were going to have to increase the size of the sensor even further… and that’s how we ended up with 41-megapixels.
A few months later, in October 2008, the initial prototyping had been done. There was enough evidence now to show this was possible, although we knew there were going to be lots of challenges ahead of us.
Many different optical designs were trialled, using different lens configurations, lens materials, lens designs etc. In the end, I think we considered around 40 design proposals. As one aspect improved, another became worse. We continuously changed and then evolved the design until we were completely happy with the balance of the various aspects.
But even then, while we knew the camera performance would be really good, we didn’t know how good. Simulations are one thing but with so much complexity involved in the image processing as the area of the sensor used changed and effecting scaling and oversampling behaviour, we never really knew that we could be 100% confident what would work well and what wouldn’t. A great deal of discussion and simulating was carried out to try and predict every eventuality, but there’s only so much you can do.
When the very first prototype camera modules became available, the excitement and anticipation of all those involved in the project was pretty extreme. Would it be as good as our simulations showed? One sample was sent to our friends at Carl Zeiss for testing around this time. A few of guys from our imaging team went to take some shots over the Pyhäjärvi lake, which lies in between cities Tampere and Nokia (yes, there really is a city called Nokia in Finland).
I remember the content of two emails still to this day. One from the Tampere team with images attached captured with the first prototype camera and another captured with a Canon digital SLR as a reference. I opened both images and viewed immediately at 100%. Initially, I thought the images were labelled wrongly. Then I also saw the email from Carl Zeiss with the results from the lab testing. It’s usual for Carl Zeiss to provide a list of comments on areas where improvements could be made. On this occasion however, the email was uncharacteristically short. Here’s a short unedited excerpt from that email: “Our lab people are VERY happy with the quality. ”
Relief!
This is, without doubt, our most complex imaging project to date. Often the ‘big idea’ has involved much discussion, but throughout the development process, as exciting as it may sound to introduce a device equipped with a 41-megapixel image sensor, our real excitement has ALWAYS been associated with the opportunities and in particular the performance this provides in its default form when shooting ‘just’ 5-megapixel images or when recording full HD video. We’ve waited a very long time to be able to do what we believe is right and break free from the years of legacies laid down behind us.
No comments:
Post a Comment