He seems to really be calling the shots!

Ben Affleck arrived back at his $20 million Pacific Palisades home this morning to workout with a trainer, just one day after Playboy model fling Shauna Sexton visited him at his Malibu rehab facility.

According to multiple reports, the Oscar winner isn't taking his recovery very seriously, and it's worrying those closest to him.

"He is trying to do the speed round rehab program and is telling everyone what they want to hear, but his team of counselors is getting very frustrated because he's not listening," a source dished to Radar.

"Everyone will be amazed if he remains in the program for another week, let alone two. No one in his life thinks that he should be getting involved with Shauna. Everyone thinks that she is using him," the source added.

We caught his ex Jen Garner going to church with his mom on Sunday, and we bet they're turning to God for some guidance on how to manage this stressful situation!