I come from this industry so I will offer some reflections (sorry, this turned into a long-form essay).
First, I have a lot of empathy for the Astrobotic team that developed the lander. To spend several of your finite years on this planet creating something, only to have that something stumble before it even really got started, is a tough pill to swallow. Many years ago I was in charge of designing and fielding a rocket prototype. It took about two years to get everything ready. Finally the day came to press the big button, and BOOM. Fireball. No more prototype. The underlying issue was a mundane defect with a vendor-supplied component. Things snowball quickly in spaceflight, and unfortunately the passing grade is 100%. After the fact, I realized that I went through a grieving process to deal with the loss of something that I had worked so hard for: denial, anger, bartering, acceptance. A harsh mistress, indeed.
Next is the question on why we should engage in space exploration when we have a multitude of problems on Earth. It's an important question. Feeding and providing the most minimal of healthcare to children in sub-Saharan Africa takes a few dollars; rockets to the Moon take billions. The first thing to realize is that this moral quandary is not unique to spaceflight. It applies to any human endeavor that does not directly pertain to providing food, shelter, and medical care -- those basic things that allow us to continue existing. Everything else -- sports, art, books, movies, some branches of science, and spaceflight -- are in the bucket of enriching our existence. I know that I am happier living in a world where a man walked on the moon and I can watch the Mandalorian, even if neither of those things is necessary. It is hard to imagine life without at least some of these unnecessary things. They are the things that make us human.
At the same time, technological progress is a difficult thing to plan, and the practical implications of a seemingly esoteric discovery can take decades and centuries to pan out. Things that initially may appear as "cool, but who cares, really" can underpin future technologies that now have impact in the real world. In the 1800's a mathematician developed Riemannian geometry. At the time this had nothing to do with anything (practically speaking). In the early 1900's Einstein took this math off the shelf, dusted it off, and realized it is exactly what he needed to describe a more accurate theory of gravity, general relativity. Here in the 2000's, the GPS used in all of our aircraft would not function without corrections for general relativity.
On a long time scale, the Earth will face an existential crisis, and at that time nothing will be more necessary than space technology. We cannot count on figuring it out when the doomsday asteroid shows up. We have to start now, as we are able.
So where to the draw the line? I do not know. I think we should push civilization forward on all fronts, while acknowledging that we need to keep trying to get the balance right.
Finally, let's talk about so-called commercial space. In one sense, commercial space has been around for a long time. There is a great multitude of commercial spacecraft buzzing around you in Earth orbit right now, providing communications services and collecting data. The first commercial satellite was launched in the 1960's and operated by AT&T. Anyone who uses communication technology, which by default includes everyone on this forum, benefits.
But nonetheless, we see talk of "new space" or "commercial space" in the news as a departure from how things were done before. In this case we are talking about manned spaceflight or scientific missions that would have previously been directly operated by NASA. For the most part, at this moment in history, this type of "commercial" space is largely government space by other means. NASA is still the customer and footing a large part of the bill. In the case of the failed Astrobotic lander, the primary payloads were NASA instruments. It is a procurement architecture intended to reduce development times and costs by reducing government involvement in day-to-day development. More failures in unmanned missions and tests are expected with this framework (think all of the SpaceX boom-booms in South Texas), which is okay so long as a robust finish line is reached at a point sooner than the old way of doing things. For the old way of doing things, please see the cost and schedule associated with NASA's Space Launch System.
While NASA is the primary bill-payer, billionaires are contributing their own cash and occasionally having some fun with it. Jeff Bezos, for instance, famously took a joy ride in Blue Origin's suborbital system. It is a funny notion that spending an obscene amount of taxpayer to send a civil servant into space is heroic, but a billionaire spending much less of his own money to do the same is somehow selfish and misguided, isn't it? The main consequence of the billionaire influx is a subsidization of launch vehicle development. This lowers the cost of putting stuff -- any stuff -- into space, as shown by the drastic reduction in launch prices since SpaceX fielded Falcon 9. This in turn lowers the costs of using space for any purpose, which includes commercial and humanitarian uses that benefit all of us normal people here on Earth.
I do not want to see obnoxious corporate behavior in space. Today that's not really an issue. The bigger issue is prohibitive spaceflight costs that limit the science, exploration, and Earth services that can be provided. A framework that reduces those costs is a welcome advancement. Hopefully appropriate treaties and regulations can keep it inside the lines.