I.e., back to Colin Grove, it would appear.

What may give an impression that the milk boy's home is elsewhere than Colin Grove is Louisa's juxtaposing of her concluding remark ("I have sent my note by him") with her statement of concern about his having "a bad time home"—which tends to suggest that the milk boy has just left Colin Grove for home. The concluding remark is likely added as a kind of afterthought.

Likely the milk boy went to Halifax earlier in the evening, crossed back over the Harbour while it was still light--and perhaps while the storm abated--and now, somewhere between Dartmouth and Colin Grove, is wending his way homeward in the darkness and the downpour.