If you say so

every time I hear someone use "if you say so" as a
response to something said, I wonder how to take it.

As a very mild reproach; not quite an insult. It means "I wouldn't take such a foolish position, but if you insist, I'll pretend to almost agree with you."