This idiom means to check a situation in a small (safe) way before becoming heavily involved in it. The expression refers to how a person checks the temperature of the water in a bath or swimming pool before putting his or her whole body in.
In business: Let’s test the waters with a small group of customers before making a large order.
In a social situation: I’m not sure if I want to join the fitness club. I’ll take a trial lesson to test the waters.