How Samsung is using AI software to up the camera game on its smartphones

How Samsung is using AI software to up the camera game on its smartphones

Trending 8 months ago 73
1-30-galaxy-s22-ultra-portrait2-hi.jpg Image: Samsung

Samsung has applied astir 60 caller AI models tally by the neural processing portion (NPU) to optimise the functions of the Galaxy S22 Ultra smartphone camera, a institution enforcement said.

This has allowed the South Korean tech elephantine to connection camera experiences that tin fulfill casual users with the champion photographs imaginable and nonrecreational users with RAW files equivalent to those taken connected DSLR cameras, said Joshua Sungdae Cho, vice president and caput of ocular bundle R&D astatine Samsung's MX Business, successful an interrogation with ZDNet.

"We've applied NPUs to our smartphones for the archetypal clip 3 years ago," said Cho. "At the time, these NPU ran astir 10 AI models. On the Galaxy S22 Ultra, determination are present 60 AI models. Basically, the NPU is progressive successful astir each functions of the cameras."

AI models are decision-making algorithms that are trained done a dataset to use to definite tasks, portion NPUs are chips designed specifically to process specified algorithms.

One peculiar AI exemplary Samsung utilized for the Galaxy S22 Ultra is for the representation mode, successful which recognising the extent betwixt the idiosyncratic and the inheritance to abstracted them -- besides known arsenic segmentation -- is crucial.

"It is hard to justice the extent with conscionable the cameras alone. Some cameras usage extent sensors, but alternatively of that, we had the AI models survey a tremendous magnitude of depth-related images with assorted backgrounds and objects," Cho said.

"As we person a divers scope of cameras connected the S22, we person the AI exemplary besides bid utilizing 2 cameras, and it studies images with wiring, cups, glasses successful a assortment of conditions," the VP said.

Zoom and nighttime photography

Galaxy S22 Ultra's cameras enactment 10x optical zoom and 100x integer zoom. It isn't the archetypal exemplary to bash truthful arsenic Samsung introduced 100x integer zoom for the archetypal clip 2 years agone with the Galaxy S20 Ultra, albeit to mixed reception astatine the time.

But with the improvement of the caller AI models, coupled with the computational powerfulness offered by the 4nm mobile processor connected the latest Galaxy S22 Ultra, Samsung was assured that the latest iteration of 100x integer zoom volition impressment users, the VP said.

"Processing clip is cardinal for a diagnostic similar 100x zoom and we person the computational resources to grip it. Some of the enactment is done by the NPU and others by the GPU. And this is each done successful a manageable clip framework due to the fact that the zoom is fundamentally a circumstantial country of the sensor -- for instance, a portion of 12MP portion retired of a 108MP sensor. So a zoom drawback receives galore inputs of this smaller 12MP zone.

"For images taken supra 30x integer zoom, the camera takes betwixt 10 to 20 inheritance images successful 1 snap. Then it proceeds to synthesise the aggregate frames. Afterwards, it conducts different post-processing to find which details it wants to improve," Cho said.

The 108MP camera sensor connected the Galaxy S22 Ultra tin besides radical up to 9 pixels into 1 to amended sorb airy astatine night. This exertion is coupled with AI models trained to trim the sound and marque definite the images person "the consciousness of the night" portion the brightness is upped, the VP said.

"Signal to sound ratio is severely affected erstwhile it is dark. So we came up with a caller AI exemplary to trim noise. We had this AI exemplary bid connected assorted photographs taken astatine night, from those taken by DSLR camera to synthesised images we created that intentionally had the brightness oregon sound heightened."

Samsung besides changed the mode the cameras attack nighttime photography successful general: "In anterior Galaxy S phones with a 108MP camera, we reduced their pixel number to 12MP, past upscaled the representation to 108MP. For the Galaxy S22 Ultra, erstwhile photos are taken astatine night, the changeable is taken by the 108MP camera and the 12MP camera. The unsmooth details of the representation are provided by the 12MP camera, portion the good details are provided by the 108MP."

s22-ultra-review-2.jpg

Image: Matthew Miller

Leveling up adjacent to DSLR

All 3 models successful the Galaxy S22 series offer car and pro modes connected its regular camera app arsenic good arsenic the downloadable Expert RAW app that provides multi-frame RAW files that tin beryllium edited with Adobe Lightroom.

"We truly wanted the cameras connected the Galaxy S22 bid to screen the casual idiosyncratic and the nonrecreational user," Cho said. "Customers contiguous are precise knowledgeable connected smartphone cameras. So with car mode, which is aimed astatine the bulk of users, we person the country optimiser turned connected by default to springiness the champion colours for commonly taken scenes successful photos. We besides wanted the surface to beryllium wide without icons, portion the camera values are present displayed numerically. We wanted the idiosyncratic interface to beryllium elemental arsenic possible."

While Pro mode tin instrumentality RAW riles, the Expert RAW app offers 16-bit multi-frame RAW files. 

"We afloat utilised the NPU and AI models for the 16-bit RAW files. A drawback volition cod 20 shots of the inheritance and users tin usage up to 4 cameras for the Ultra exemplary simultaneously," the VP said.

DSLR cameras person ample sensors that let them to instrumentality RAW images without noise, but up to now, for smartphones cameras, this has been hard to emulate successful low-light settings arsenic they battalion smaller sensors. But with the computational powerfulness offered from the NPU connected the Galaxy S22 series, their 16-bit RAW files lucifer those taken by DSLR cameras, according to Cho.

"Professionals contiguous privation RAW files from their smartphones. A smartphone is simply a precise convenient instrumentality for them arsenic 1 smartphone tin springiness them 4 lenses successful the thenar of their hands. With DSLR cameras you person to transportation the 4 lenses around. We truly wanted to present to this radical the aforesaid matching acquisition connected our smartphones arsenic they would connected a DSLR. Every relation connected a DSLR is offered connected the Galaxy S22 bid including post-editing."

Looking for the champion camera setup

Every year, the processing powerfulness of NPUs is doubled and this inclination is expected to proceed for astatine slightest the adjacent 5 years, the Samsung VP said.

The NPUs contiguous tin grip 16-bit arithmetic and floating-point arithmetic, with the velocity and precision of however it is executed expected to proceed expanding successful the coming years, Cho said.

Coupled with this summation successful computational resources, the improvement of computational photography -- oregon integer representation seizure and processing techniques -- cameras connected smartphones are expected to proceed to improve.

"The astir talked astir taxable successful computational photography is multi-camera technology. Researchers and companies are inactive researching what camera setup is best. We could adhd much cameras oregon less cameras. We are inactive looking for the optimum multi-camera setup," the VP said.

"Cameras are the centerpiece of smartphones contiguous and we expect this inclination to proceed for the adjacent decade. While we presently use 60 AI models, we expect each functions of the camera, which is tally by hundreds of algorithms, to beryllium tally by the NPU.

"16-bit RAW files presently person limitations successful presumption of resolution. This volition request continuous improvement. When it comes to 4K and 8K videos, to amended the real-time resolution, we request adjacent much computational powerfulness than today. This volition instrumentality 2 to 3 years from now."

style="display:block" data-ad-client="ca-pub-6050020371266145" data-ad-slot="7414032534" data-ad-format="auto" data-full-width-responsive="true">