For a whole host of reasons. In summary, quality of life for those remaining is going to crater, together with some form of social collapse.

1. Most social insurances (e.g. pensions and welfare) depend on young, healthy, working people paying for those in need. As the population pyramid gets inverted, eventually this will become completely unsustainable. Meaning those who are young now will not be able to benefit from a pension in the future.
2. Health care costs are going to soar to unsustainable levels. To some extend, this has already happened. Again, old people tend to require health care a lot more frequently - even permanently, usually- than young people. As the population pyramid gets inverted, this means ever fewer young people have to care for ever more sick people. As an example, my country estimates that by 2050 we’ll need to spend 40% of GDP and 1 in 3 working people on health care if we want to keep the service level at today’s standards. That’s of course completely unrealistic. To some extent this is already starting to deteriorate.
3. Ever fewer people will have to maintain essential services. Think sanitation, sewage, construction, rail services and so on. Again, unsustainable.
4. The gerontocracy will mean society will become increasingly inflexible, rigid, and stuck in the past. Young people drive change, old people like to keep things as they are. Opinions don’t usually change. Instead, they die one funeral at a time.
5. The economic challenges caused by an aging population will require tough choices. But with the gerontocracy, such choices will likely not be made. Or they will only be passed on to next generations (who get no say in the matter, as they will be too small a voting bloc). Ultimately this will necessarily lead to some form of social collapse.