It is unclear whether they go to Mrs. Allen's or to Mrs. Brinley's instead.

If to the former, then it seems that in referring to Mrs. Brinley, Louisa is simply explaining how she received the invitation to tea in the first place (i.e., while she was picking berries). If to the latter, then it seems that when Louisa says she "went to Mrs. Allen's," she really means "set out for Mrs. Allen's." On the way, she would have bumped into Mrs. Brinley, who would have persuaded her to go to her place instead of to her mother's.