#1. You have to look at.. ahem... the hamster's genital area. Well first pick up your hamster (go check out my handling post if you need help on that) and look below your hamster. There genital area is located just below their stomach/under their tail.
#2. If it's a male you will see...ahem... a penis awkward discussion but the males will have more of a bulge, that is visible from behind. (You also will see their testicles) The males genital is just above their bum hole/anus.
#3. For a female if you pick her up you will see nipples/breasts down from her mid chest to hips. You will also see no bulge underneath. You will just see her lady parts. Her breasts are used to feed her young. Try your best when handling a female hamster to avoid her breasts only to avoid pain for her.
-Sorry if this was slightly disturbing for you in any way but this is probably the only technique you can use to tell the gender between a male and female hamster.