These days, self-publishing first isn’t the interest-sinker it used to be with editors. If a self-published books does really well (say, selling 30,000-40,000+ copies) then it can impress editors and spark interest. Huzzah for that, of course. But if it hasn’t sold well (which is, honestly, more often the case with self-published fiction than not) but an editor likes the book and thinks she has a bead on its market, the editor will acquire your book and just have you remove the self-published edition from the market. So no, self-publishing with less than stellar results wasn’t shooting yourself in the foot. Your friend is right: Submit.