The prior art discloses gaze correction by identifying eye locations, extracting SIFT descriptors as local image features, and applying morphological transformations to replacement eye imagery. However, it does not disclose multi-view images, deriving a single feature vector from plural local descriptors, or using that feature vector to look up reference displacement vector fields. The prior art uses a database of replacement imagery rather than displacement vector fields looked up by feature vectors.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | Detailed Description, [0072] (0073. Referring now to FIG. 1, the method begins at 10 and continues to 11 at which image 100 is processed to extract locations of one or more of the eyes over image 100. The locations of the eyes can be extracted using any known image processing technique for automatic extraction of features from an image. Detailed Description, [0120] The procedure continues to 72 at which the location of each eye is extracted from the respective image of the set. Detailed Description, [0122] The procedure then continues to 73 at which the respective image is processed so as to extract image features pertaining to the eyes in the image. |
FIG. 1 | |
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
The prior art discloses gaze correction of eye regions in images, including detecting eye regions, computing displacements for eye outer points, and warping the eye texture. However, it does not disclose multi-view images, feature vectors derived from plural local image descriptors, or lookup of reference displacement vector fields using such feature vectors.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | Detailed Description, [0068] (0070. In step S320, the controller 160 detects (or extracts) a facial region and an eye region from the user image. Detailed Description, [0068] Referring to FIG.3, in step S310, the controller 160 receives an input of a user image from the camera 150, or reads a user image stored in the memory 130. 10 Although the method is described with reference a single user image, the method is applicable to each of a plurality of images that are sequentially input or read. |
||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
US9538130B1 discloses gaze correction by identifying eye regions in video frames and substituting them with stored forward-looking eye portions. However, it does not disclose multi-view images, feature vectors derived from local image descriptors, displacement vector fields, or lookup of reference displacement vector fields. The approach is fundamentally different: substitution-based rather than transformation/displacement-based.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | Detailed Description For example to modify a frame to make a participant appear to be looking forward, the decomposer software module 118 may identify (i) a portion that includes a left eye and a portion that includes a right eye or (ii) a portion that includes both eyes. Detailed Description For example, a classifier or other machine learning algorithm may be used to identify and extract portions of the captured Video frames that include the participant's eyes. Detailed Description As another example, a first portion may include a left eye of a participant, a second portion may include a right eye of a participant, and a third portion may include a remainder of a face (e.g., excluding the eyes) of the participant. |
||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
US9384384B1 discloses adjusting facial representations in images using a processor and involves face/feature identification, but does not specifically address multi-view images, gaze correction, eye patch identification, feature vectors from local image descriptors, displacement vector fields, or lookup of reference displacement vector fields. The prior art operates on a fundamentally different approach using manifolds for face model generation rather than the claimed feature-vector-to-displacement-vector-field lookup approach for gaze correction.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | – | ||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
CN107534755B discloses gaze correction in video frames of a user's face, which broadly overlaps with the preamble concept of adjusting images to correct gaze. It also involves identifying/tracking the user's face and eyes. However, it does not disclose multi-view images, feature vectors derived from plural local image descriptors, lookup of reference displacement vector fields, or transforming image patches using displacement vector fields. The core technical approach of the target patent (descriptor-based feature vectors, displacement vector field lookup and application) is entirely absent from this prior art.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | – | ||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
US8467627B2 discloses detecting eyes in a face area and performing warp processing on an image based on detected eye positions. However, it does not disclose multi-view images, gaze correction, feature vectors from local image descriptors, displacement vector fields derived by looking up reference data, or transforming eye patches in accordance with displacement vector fields. The prior art's warp processing is a simple non-linear warp (e.g., fisheye lens filter) centered on detected eyes, not a gaze-correction technique using learned displacement vector fields.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | – | ||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
US9633250B2 is directed to facial landmark localization using globally aligned regression. It deals with face images, feature extraction at landmark locations, and regression to update landmark positions. However, it does not address multi-view images, gaze correction, eye-specific image patches, displacement vector fields for image transformation, or lookup of reference data comprising reference displacement vector fields. The overlap is limited to general concepts of processing face images with a processor and extracting features.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | – | ||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
JP5645450B2 relates to multi-view image processing using multiple cameras from different viewpoints, but it is focused on viewpoint interpolation and blur processing for smooth virtual camera movement. It does not address gaze correction, eye detection, feature vectors from local image descriptors, displacement vector fields, or any lookup-based transformation for correcting eye regions. The only overlap is the general concept of processing multi-view images.
| Element ID | Element Text | Citation | Figure | Comment |
|---|---|---|---|---|
| 20-E1 | A method of adjusting multi-view images of a head to correct gaze, the method comprising: for each image, using a processor to identify image patches containing the left and right eyes of the head, respectively; | – | ||
| 20-E2 | in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, using the processor to perform the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; | – | ||
| 20-E3 | and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; | – | ||
| 20-E4 | and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. | – |
| 청구항 | 공격 유형 | 근거 문헌 | 커버된 Elements | 미커버 Elements | 난이도 |
|---|---|---|---|---|---|
| Claim 20 독립항 | §103 조합 | US9335820B2 + EP3429195A1 | 20-P (EP3429195A1), 20-E1 (US9335820B2) | 20-E2, 20-E3, 20-E4 | 상 |
20-E2 (복수의 로컬 이미지 디스크립터로부터 단일 특징 벡터를 도출): 컴퓨터 비전 분야에서 SIFT/SURF 등 로컬 디스크립터를 BoVW(Bag of Visual Words), VLAD, Fisher Vector 등으로 집약하여 단일 특징 벡터를 생성하는 기법을 개시한 선행문헌 조사 필요 (특히 눈·얼굴 패치에 적용한 사례)
20-E3 (도출된 특징 벡터를 이용하여 참조 변위 벡터 필드를 룩업하는 구성): 이미지 워핑/모핑 분야에서 특징 벡터 기반 유사도 매칭으로 사전 저장된 변형 필드(deformation field / flow field)를 검색·보간하는 기법을 개시한 선행문헌 조사 필요 (예: example-based face synthesis, flow field retrieval 관련 논문/특허)
20-E4 (도출된 변위 벡터 필드에 따라 이미지 패치를 변환하여 시선 보정을 수행): 변위 벡터 필드(displacement/optical flow field)를 눈 영역 패치에 적용하여 시선 방향을 변경하는 구체적 구현을 개시한 선행문헌 조사 필요 (예: "DeepWarp", Ganin et al. 2016 "DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation" 등 학술 문헌 포함 — priority date 2016-01-05 이전 공개 여부 확인 필수)
종합 평가: 현재 확보된 선행문헌만으로는 청구항 20의 핵심 기술적 특징(20-E2 ~ 20-E4)을 커버할 수 없어 무효화 난이도가 상이다. 특히 "로컬 디스크립터 → 특징 벡터 → 참조 변위 벡터 필드 룩업 → 패치 변환"이라는 일련의 파이프라인을 단일 또는 소수 문헌으로 개시하는 선행기술의 추가 발굴이 필수적이다.