I don't like making decisions about paint

Picking paint colors for our house is stressing me out. 

First we had to decide what the general color scheme was going to be.  I know what colors I like, but I am worried about flow.  If I get too many colors going on one floor, it could look disjointed and weird.  If I use too much of the same color, it might look boring.  And I really don't want all neutral walls; I've had enough of that from apartments. I need color in my life.

Once we talked it over and vaguely agreed on colors, James and I happily brought home paint chips to tape up on the walls, only to discover that we both have very different opinions on what shades look good.  He likes dark, I like...less dark.  He prefers more primary colors, I prefer muted colors. 

Then, once we hash it out and finally decide on a couple of shades that we can both live with, I go to Lowe's and get those little sample jars of paint, just to see what it looks like on the wall.  I have done this three or four times now.  Every single time the paint color looks completely different than the swatch.  I even tried the same color paint in the kitchen and again in the living room and they look completely different from each other. 

I don't know if it is the lighting or the color that is already on the wall or what, but I don't understand how the same paint can look so greatly different in two different rooms.  So now I'm wondering if I need to prime the whole house first, even on the more neutral walls.  But then if I do, the paint that we have picked might look different and then it's not what I wanted.

Bah!  It's not supposed to be this hard.