The rules are short, and most of the anxiety comes from not knowing them. Here they are.
The bath is for soaking, not for cleaning. You sit on a low stool at a shower station, wash properly, rinse every trace of soap off, and only then step into the bath. The water in the tub is shared and nothing goes into it but clean people.
No swimsuit. Baths are separated by gender, marked with a red curtain for women and a blue one for men, and the characters are 女 for women and 男 for men. Almost nobody is looking at you.
You get two towels: a large one you leave in the changing room, and a small one you carry. The small one is for washing and for drying off afterwards. In the bath, people fold it and rest it on their head or on the rim. Putting it in the water is the one thing that will get you a look.
Tie your hair so it stays out of the water. Take off silver jewellery before a sulfur bath, because the sulfur will tarnish it, sometimes permanently.
A ryokan is not a hotel with a bath attached. The standard booking is one night with two meals: dinner on the evening you arrive and breakfast the next morning, both often served in a set course at fixed times. Check-in is usually from 15:00 and dinner might be at 18:00 sharp, so arriving late costs you the meal you already paid for. Rooms are commonly quoted per person, not per room.
You do not have to stay overnight. Many baths accept day visitors under the sign 日帰り入浴, usually for a few hundred to a couple of thousand yen, and often only during set afternoon hours.
Worried about tattoos? That has its own page: going to an onsen with tattoos.