Alpha mask background trick
I've seen this problem more than once on CG forums: somebody has made a nice alpha masked texture, but the blending looks a bit odd. Lets take this leaf for example (colour texture and alpha shown side by side):
A nice clean alpha, should look great in 3D. But look at what happens when it's applied on a plane in your favourite 3D application:
At first view it looks nice, but when you zoom in you can see a nasty black halo around the edge of the leaf. It's not looking too good now, but when rendered or used in a game engine it'll probably look much worse.
Luckily the solution is pretty simple: just fill the background with the same colour as the leaf, so when the alpha blend the polygon to transparent the colour doesn't change:
Now the transition is perfectly smooth:
Pretty easy, but what about texture where there is a more of color variation? (apologies for the hideous texture)
For the border to blend nicely, the background needs to be blue in one spot, orange in another, and in yet another spot it should be green! You could paint this by hand, but that would be a lot of work, and boring work at that. Luckily there is a quick way to do this.
First, make a copy of your colour texture as it will be masked. (If you have a lot of layers, you can do this without flattening your document by CTRL-clicking on your alpha mask, and then pressing CTRL-SHIFT-C. This is the shortcut key for 'Copy Merged', a copy of the selected canvas as you see it, so all the layers combined.)
Paste this copy in a new layer.
Now apply a Gaussian Blur of about 2 pixels on this layer:
Copy the layer by dragging it on the 'new layer' icon:
And then merge the layer down by pressing CTRL-E. Repeat this process a couple of times: copy, merge down, copy, merge down...every time you do this you will see the layer expand and get less transparent around the edges. Four or five times will probably be enough to make your image look like this:
When the edge is no longer transparent all you have to do is make a copy of your original background and layer it on top of the blurred version to make it sharp again:
As you can see the edge pixels of the leaf basically repeat themselves around the siloutte of the alpha: wherever the leaf is orange, the background is orange, wherever the pixels are blue the background is blue. No more transparancy problems! It might look like a bit of work, but it should take no longer than 15 seconds to do this, and you could probably even automate it by recording an Action.
Copyright 2007 - Marcel Vijfwinkel