At the time of writing The Lord of Rings, Tolkien had already completed a body of work that many writers would do well to equal in a lifetime. Tom himself had appeared in print as early as 1933 (though the collection The Adventures of Tom Bombadil did not appear until 1961 - and in fact Tom only appears in the first two of these sixteen poems). What's more, the Silmarillion was already well developed (though much of it as it existed then would be unrecognisable to readers familiar with the published version, a great deal of its narrative was already in place).
It seemed at that time that the Silmarillion would never be published, and so Tolkien felt free to use names from that work in his sequel to The Hobbit: Glorfindel is the most famous example, but the names Gildor, Denethor, Boromir, Minas Tirith, and many others besides, all appear in both works, referring to different characters and places.
Tom must also have been part of this process, but in his case, his entire character, rather than just his name, seems to have been transplanted into the emerging Lord of the Rings. Tom's appearance in the early chapters is natural - Tolkien at that time seems to have envisaged the work as a children's book, a sequel to The Hobbit following the same style, and Tom certainly would not have seemed out of place. As it grew, though, the world of The Lord of the Rings began to merge with that of The Silmarillion. Here the difficulty seems to have arisen - a character like Tom, though he fits easily into the unconstrained story-telling style of The Hobbit, doesn't have an obvious place in the detailed universe of The Silmarillion.
Though Tom's insertion into the nascent Lord of the Rings might be viewed (at least in a sense) as 'accidental', it is certainly no accident that he remained there. Tolkien reviewed and revised the book with his customary meticulousness - it is inconceivable that the character of Tom Bombadil would have stayed in place if Tolkien didn't see him, in some sense, 'fitting' with the rest of the story. In Tolkien's own words:
'...I kept him in, and as he was, because he represents certain things otherwise left out.'
~The Letters of J.R.R. Tolkien, No 153, dated 1954