Browns pretty much all work. Dark brown is usually enough contrast. Brown can look significantly better with texture, though, so consider that.
Tan/Khaki/Camel works. I prefer this color for wool over chino, although the chinos are much more common.
Olive green should work.
If you want to wear white pants, the combination works.
Dark greens, dark burgundies, even dark purples and burnt oranges if you’re daring… Those are all fun options.
Maybe light wash denim, if you want to try dressing it down. Should be enough contrast from navy blue.
There’s a reason everybody told you to get a navy blazer.