Detail mattes and ID maps (suggestion) #106

New Issue

claunia · 2026-01-29T21:42:13Z

claunia commented

2026-01-29 21:42:13 +00:00

Originally created by @KeygenOld on GitHub (Nov 12, 2021).

Would be cool if we could define a "detail matte" to help the algorithm figure out which areas should be more detailed and which should be blurry and out of focus. This could be a secondary grayscale image. Black would be lowest detail (blurred), 50% gray would be neutral areas that are ignored, white would be the highest detail, and gradient would fall between.

This doesn't need to be a very detailed image, and could just be a sort of suggestion to the software and painted using soft brushes in photoshop or gimp, to define which areas should not be ignored due to image degradation.

Would also be good to be able to set a flag to create a secondary image—this way we could get a render that uses it and one that doesn't, in case we need to blend the two, i.e. --output_detail_image to render both. Some people may even find a delta image useful, but not necessary for myself.

This could be useful for:

very old photos that have shallow depth of field or lack detail
generating lost skin texture in soft or out of focus faces
keeping background out of focus in grainy photos
enhancing detail in clothing
make sure hair does not get too out of focus near the background if it's not supposed to be
ensure hair in shadow is given detail
for making stylized images
to fake depth of field

An alternate idea would be to use a "heat matte". These are used in 3D for defining various things, such as the amount that skin geometry should slide over muscle or how much a body should jiggle. These could be grayscale images, but may be RGB images, or layered RGB images, so you can use color to define various parameters. This is probably too complicated to be bothered with, but suggesting it anyway as food for thought.

For GFPGAN, a heat map could work as follows:

Black could be neutral, and means the algorithm behaves normally and ignores it
Blue could be used to define blurry/out of focus areas that are in the background far from subject
Red or White to define foreground areas in the face that need more detail
Green to define secondary things that need detail (like clothing)
Yellow to define foreground objects that should be out of focus, but not as much as background
saturation/opacity of color would define intensity

If you wanted to go really crazy, you could use different colors to define parts of the face (hair, eyes, nose, mouth, ears), for images that are very damaged.

ID Maps or Layered Masks
Another useful feature would be to output an ID map so that parts of the image could easily be masked in post (photoshop, etc). This is something else that comes from CGI. Modern render engines can output a render of various colors that can then be used as masks in post by plug-ins that are made to recognize them and avoid aliasing so you can color correct and composite things easily. Outputting an ID map that defined background, face, and the rest of the body could be very useful for final adjustments. If not, a simple layered TIFF file with 3 layers of solid colors and alpha channels would suffice and be more accessible to people used to 2D graphics.

Not sure if it's worth your time to do any of this, but just some ideas that came to mind while experimenting with the tools so figured I'd share just in case you thought they were of some benefit to your research. I could personally use them for the images I'm working with.

Originally created by @KeygenOld on GitHub (Nov 12, 2021). Would be cool if we could define a "detail matte" to help the algorithm figure out which areas should be more detailed and which should be blurry and out of focus. This could be a secondary grayscale image. _Black_ would be lowest detail (blurred), _50% gray_ would be neutral areas that are ignored, _white_ would be the highest detail, and gradient would fall between. This doesn't _need_ to be a very detailed image, and could just be a sort of suggestion to the software and painted using soft brushes in photoshop or gimp, to define which areas should not be ignored due to image degradation. Would also be good to be able to set a flag to create a secondary image—this way we could get a render that uses it and one that doesn't, in case we need to blend the two, i.e. `--output_detail_image` to render both. Some people may even find a delta image useful, but not necessary for myself. **This could be useful for:** - very old photos that have shallow depth of field or lack detail - generating lost skin texture in soft or out of focus faces - keeping background out of focus in grainy photos - enhancing detail in clothing - make sure hair does not get too out of focus near the background if it's not supposed to be - ensure hair in shadow is given detail - for making stylized images - to fake depth of field --------------- An alternate idea would be to use a "heat matte". These are used in 3D for defining various things, such as the amount that skin geometry should slide over muscle or how much a body should jiggle. These could be grayscale images, but may be RGB images, or layered RGB images, so you can use color to define various parameters. This is probably too complicated to be bothered with, but suggesting it anyway as food for thought. For GFPGAN, a heat map could work as follows: - Black could be neutral, and means the algorithm behaves normally and ignores it - Blue could be used to define blurry/out of focus areas that are in the background far from subject - Red or White to define foreground areas in the face that need more detail - Green to define secondary things that need detail (like clothing) - Yellow to define foreground objects that should be out of focus, but not as much as background - saturation/opacity of color would define intensity If you wanted to go really crazy, you could use different colors to define parts of the face (hair, eyes, nose, mouth, ears), for images that are very damaged. ----------------------- **ID Maps or Layered Masks** Another useful feature would be to output an ID map so that parts of the image could easily be masked in post (photoshop, etc). This is something else that comes from CGI. Modern render engines can output a render of various colors that can then be used as masks in post by plug-ins that are made to recognize them and avoid aliasing so you can color correct and composite things easily. Outputting an ID map that defined background, face, and the rest of the body could be very useful for final adjustments. If not, a simple layered TIFF file with 3 layers of solid colors and alpha channels would suffice and be more accessible to people used to 2D graphics. Not sure if it's worth your time to do any of this, but just some ideas that came to mind while experimenting with the tools so figured I'd share just in case you thought they were of some benefit to your research. I could personally use them for the images I'm working with.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: TencentARC/GFPGAN#106