How should I move and process this data?
Start from what the consumer cannot currently do, not from what the source happens to produce. Reasoning in the other direction is how a platform fills with datasets nobody uses — and every leaf here states what its recommendation costs, because a recommendation without a cost is an advertisement.
The flagship decision. Start from what the consumer needs, not from what the source produces — reasoning in the other direction is how platforms fill with datasets nobody uses. Every answer names its trade-offs, its failure modes and something simpler you should have ruled out first.