Detailed analysis of GPUImage (eleven) + beauty face recognition

Preface

A group of friends made a demo with Kun Jun beauty recording and Xunfei offline face recognition SDK, function is to record video, for beauty, and face recognition and placement. But encountered a problem:
recording process can be made for face recognition, but also has a beauty effect;
video recording, a beauty effect, but there is no map;
in the process to help find bug, found the code is slightly complicated, not easy to learn.
so, take the time to pull out the core code, made this demo.

The results are as follows:

Detailed analysis of GPUImage (eleven) + beauty face recognition

text

Core logic

The flow chart of demo is as follows:

Detailed analysis of GPUImage (eleven) + beauty face recognition

in GPUImage in detail (three) – based real-time beauty filters on the introduction of the IFlyFaceDetector, using GPUImageUIElement to draw the map of face recognition, and the image of beauty after the GPUImageAddBlendFilter (GPUImageBeautifyFilter) and texture (GPUImageUIElement) combined to GPUImageMovieWriter file.

Face recognition

IFlyFaceDetector
IFlyFaceDetector is a local face detection class provided by iFLYTEK, can face detection, video stream detection function.
initialization code is as follows

Self.faceDetector = [IFlyFaceDetector sharedInstance]; if (self.faceDetector) {[self.faceDetector setParameter:@ "1" forKey:@ "detect"]; [self.faceDetector setParameter:@ "1" forKey:@ "align"];}

Demo will use IFlyFaceDetector to identify the interface NSData

* / * * frame detection in video frames face * * @param frame * @param width video frame data of video image frame width * @param height video frame image @param image dir * * * @return direction JSON format face array, the face is not detected or a null * / - (NSString*) trackFrame: (NSData*) frame withWidth: (int width) height: (int) height direction: (int) dir;

CanvasView
CanvasView is a group of friends to provide demo in the mapping class, you can head, eyes, nose, mouth, face mapping, the demo will use the headMap header map.

@property / head map (nonatomic, strong) UIImage * headMap; / / @property (nonatomic, strong eye map) UIImage * eyesMap; / / @property (nonatomic, strong map nose) UIImage * noseMap; / / @property (nonatomic, strong mouth map) UIImage * mouthMap; / / facial texture (nonatomic, strong) @property UIImage * facialTextureMap;

GPUImage correlation

GPUImageAddBlendFilter
class GPUImageTwoInputFilter used to merge two images, the formula is as follows:

Float R; / / red color component if (overlay.r * base.a + base.r * overlay.a * > overlay.a = base.a) {r = overlay.a * base.a + overlay.r * (1 - base.a) + base.r * (1 - overlay.a);} else {r = overlay.r + base.r;}

GPUImageUIElement
GPUImageUIElement GPUImageOutput class, as the source of the response chain.
through the UIView to the CoreGraphics image, and through the glTexImage2D binding to the outputFramebuffer specified texture, and finally notify the targets texture ready.

The role of demo is to turn CanvasView into texture and pass it on to GPUImageAddBlendFilter.

Problems encountered

1, the map can not appear in the recorded video

Start the group of friends to provide demo preview, normal, video recording does not map;
response chain code, find the realization of the code there is a problem that is not the same
preview frames and write video frames, GPUImageUIElement output is directly to the merger of the filter, the merged image output directly to the writer to write the file; screen map preview is because the canvasView is directly to the view layer in addsubview.
doubt that the texture of the GPUImageUIElement is empty. By checking the texture of the GPU, GPUImageUIElement corresponds to the texture of the ID for the normal texture, to exclude this problem. Check the output
beauty filter, also normal.
check merge filter output, found that the map disappeared.
is positioned to merge the filter problem, check the shader code, normal.
check initialization code, find the problem:

The group of friends merges the filter of the mix=0.0; causes the merged filter to take only the first image.

Summary, in the process of looking for bug, because demo is more complex, spent more time familiar with the code through the Xcode tool, you can quickly locate the majority of GPUImage problems.

2, the map does not move with the face

Test the process of the demo, there have been fixed fixed the situation of the map.
by checking the output of face recognition, determine the output of face recognition is normal;
check canvasView update, found the problem:
canvasView no update.

The solution is to add canvasView to the view layer.
but do not know whether the impact of [self.viewCanvas setNeedsDisplay].

summary

Demo here, the code is shorter.
because it is the recognition of each frame, so the higher consumption of CPU.
if it is a practical application, you can consider the 3~5 frame to do a face recognition.
there is another simple way of thinking: the input from the camera into a video, the video frame by frame face recognition and mapping it into the video.