Yup, Hydrogen peroxide, diluted, is the way to go. I used to have a job where we were in the water -often- and we'd get swimmer's ear every other month or more.

Just lay on your side, drip the diluted peroxide in your ear from the peroxide cap and let it bubble away. When the bubbling subsides, roll over (with a tissue under the first ear) and repeat the process with the other ear.

Alcohol and anything that contains it is right out. It dries out the ear and could make things worse.