Looks like Nicole Richie wears the pants in the family ... and baggy ones at that!

The mom of two, wearing loosely-fitting red pants and a sheer black top, hit up Pink Taco in Century City with her two kids in tow for an afternoon bite.

We're told the Fashion Star mentor, who enjoyed lunch with two other moms and their kids, munched on the signature chips and salsa. Sounds delish!

Too bad she didn't take anything "to go" for hubby Joel Madden.