Proper masking is time consuming. Let's begin with that. Any attempts to shortcut through using filters or some other automated masking technique won't look as good, or be as usable as a plain old handmade mask. I'm sure someday this won't be the case, but for now just do it right the first time and you'll never have problems down the road with a mask that worked for one purpose but now doesn't work for another.
Here's the original image I'll be masking (yes, its crap quality, but that's fairly common)...
Generally, masking should begin by cutting a path. Go to your pen tool and make sure its set as follows (rubber band on, path mode set to "exclude overlapping path areas")...
Now to draw the path, just start anywhere on the edge and begin creating your path. Keep in mind that you'll need to cut a few pixels in on the edge, this will give you a nice clean edge. Cut your path right at the boundary and you'll have leaks and halos all over the place when you're done. Your path should begin looking like this...
Since I'll need to hand paint the hair in, I'll just cut in on this area with my path and paint the hair later...
Once you've finished and closed the path, you need to load it as a selection (either contextual menu drop-down as shown, or click on the path icon/name while holding the command key down)...
Now we'll create a group and place our image layer into it, while creating a background color layer that's something fairly mid-to-dark in tone and of different color to the image, so we can make sure there are no leaks anywhere (in this case I used a brown, as that's similar to what the image will be placed on later)...
Add your mask to this layer group, and you'll get the following, which as you can see is perfectly sharp and feels "cut-out"...
To solve this we'll blur the mask a bit, just type in the amount that makes the edge look natural in this case, .5 pixel blur)...
Before going any further, spend a little time moving around the edge and make sure there are no leaks or other visual issues. if you find any, fix them before moving on.
Now this image has a hair mask I need to make, so I'm going to paint all this in with my Wacom tablet. The brush settings for what I feel make the best hair strands are as follows...
Brush set to 0% hardness, 10-20% spacing maximum, smoothing on
Shape dynamics - Size jitter tied to pen pressure, angle jitter off
Opacity and Flow Jitter set to pen pressure
Now I paint, basically one hair at a time, from the outside (where the hair strand ends) into the main body of the mask, until I've created a natural looking and clean hair mask. It doesn't need to follow the real hair perfectly (in fact in this instance his hair was way too tall and would have looked ridiculous), it just needs to look clean and natural. Here's the final hairline and mask...
And the final image looks like this against my brown background...
Of note: always put your masked image into a group, and apply the mask to the group. Any and all edits for this image then go inside that group. You always want a single master mask like this to define the entire edge of the image, doing it any other way will inevitably lead to issues down the line. In fact, working in this way will actually prevent a lot of common problems designers and retouchers run into using inferior organization methods.
Also, if you work in this way you'll now have a nice clean clipping path to use in the event you need it. Now since I had a hair mask on this image its not useful in current form, but if this were a product shot or something else with all hard edges, I'd have a finished clip. And it always pays to have a clean, resolution independent masking source for future use, you never know when you might have to blow this thing up 500% and your old mask becomes to soft or rasterized to use at the new size. Simply load the clip and soften it again for the new size, and you're ready to go. If you don't have the clip, you have to start over.
Additionally, if you wanted to add a fade mask to this, do it this way. Create a new layer group and nest the player group you just created into it...
Add a mask to it and use the gradient tool to create your fade. If there's any banding present, use the techniques in noise part 2 to remove it. The result is this mask...
Which creates this fade...
The important thing to note here is again, use layer groups to create masking. Here I have two levels of masking, which allows me to create more complicated transparency effects without ever doing destructive edits to my primary image mask. You can create a 5 group deep hierarchy in Photoshop, use this to your advantage. Never apply masks to a layer or group of layers, and never modify a mask by applying a fade to it that you might need to alter later on (or remove entirely).