Well, sure. But there are course buoys, and the savvy open water swimmer will take advantage of them to rocket past a clustered group, shake a persistent drafter or begin a final sprint.
The Corkscrew Open Water Turn has emerged as one of the fastest ways to get around a buoy. It is a much tighter (hence faster) turn than a conventional turn where you simply continue to swim on your front (see the red path on the photo).
The key to the turn, described in more detail in the article, is to put your body through a complete rotation (front/back/front) through a sequence of three strokes. It is something that benefits from practice, so give it a try!
How about cable courses courses like the one at Chris Greene Lake, where you end up swimming in the opposite direction as initially? In that case, you can do two sequential corkscrews to carve a tight path around the buoy. This is definitely something you want to practice so that you don’t end up “stuck” on the buoy, but it can really gain you some time over your competitors.