You are not alone in drawing those conclusions. Philosophers and physicists have fallen into the same mental trap. Adolf Grünbaum wrote this paper (entitled The Pseudo-Problem of Creation in Physical Cosmology) in which he addresses the issue of wondering where things came from before the Big Bang. I myself recently read it and found it a solid piece of work that is also readily accessible to the lay-man (I am just recently learning and catching up with the current cosmological discussion). I am not sure how much of our understanding has changed since he wrote this paper (e.g. the estimated age of the universe has changed since he originally wrote the paper), but he does offer new alternatives to thinking about the time singularity presented in the Big Bang model.
I will take this time to directly quote a piece from the paper that represents his argument that is relevant to your qusetion, however reading the entire paper is highly encouraged.
"Case (ii). This subclass of big bang models differs from those in Case (i) by excluding the mathematical singularity at t = 0 as not being an actual moment of time. Thus, their cosmic time interval is open in the past by lacking the instant t = 0, although the duration of that past interval in years is finite, say 12 billion years or so. But just as in Case (i), no instants of time exist before t = 0 in Case (ii). And despite the equality of finite duration of time intervals in the two models, the crucial difference between Case (ii) and Case (i) is the following: In Case (ii), there is no first instant of time at all, just as there is no leftmost point on an infinite Euclidean line that extends in both directions. And in both Case (i) and Case (ii), the non-existence of time before t = 0 allows that matter has always existed, although the age of the universe is finite in either case. And this assertion is true because, in this context, the term "always" refers to all actual past instants of time.
Nevertheless, even in Case (ii), the finite age of the universe has tempted some people to make the tacit false assumption that there were moments of time after all before the big bang, an assumption
incompatible with both models. And once this question-begging assumption is made, the door is open for all the same illegitimate, ill-supposed creation questions that I undermined à propos of Case (i)."
(Adolf Grünbaum, Philosophy of Science, Vol. 56, No. 3, Sept. 1989, pp. 373-394)
I hope this helps in some way. Cheers.