Another item that must be in any man's closet is chino pants. Chinos are not khakis okay? Let's get this straighten out right now. Chinos are more tapered, lighter fabrics and never pleated.
Khakis are more for outdoor types, it's thicker and more durable. Chinos has become an alternative choice instead of jeans for casual menswear.
Yes, they do come in many colors, so it's not only brown. So what color should you have? Obviously your first pairs will be brown. It goes with most colors shirts and shoes. I would suggest you get brown, black and navy blue as your base chinos, same way you would pick your sports jackets. Other colors can be added on later.
IF you also have plaid jackets in brown, double check that too, so you don't have too many browns chinos in your closet.
When you pick brown chino, check out the color of your sports jacket first, I would go match it, then buy another darker shade of brown to wear it with my navy jackets and vice versa but you can do whatever you like to your taste.
Personally, I prefer to wear darker jacket with lighter color pants not the other way around when talking of casual look in the summer and darker shade or match the colors of jackets for Fall look.