I guess this is the thing. We're not just talking about utility in a finite combat situation, but in a grand simulation sense. (Yeah, I know this is pretty ambitious -- Dwarf Fortress, The Sims, CK2, Prison Architect, and RimWorld as references.)
If I effectively multiply the number of actions by the number of characters, that's a LOT of actions.
I guess the best way to cull some of the actions is to only consider killing a finite list of "enemies". Or only consider killing the top 5 people I dislike. Similarly, with food I might consider only sources within range, plus whatever I have at home.

Not every action is valid in every game state. Sort your calculations so that the most likely to fail are checked first. This gives you an early-out property that is extremely valuable. For example, I may have a high utility for killing an enemy, but if that enemy is 5km away, the utility is zero. So check his distance first, and bail out of the calculation as soon as you notice that the utility of attacking is very low/zero.

The Building a Better Centaur talk has more details on similar tricks you can do.

If you DO want to have a high-level "goal" of sorts, you can determine that at a slower interval and set a flag. Then lower-level behaviors can be switched on or off (or prioritized with a utility function) based on the state of that flag. I'm just confused by what your idea of "satisfying needs" is here. That's not a high-level goal... that's a pretty immediate one in many cases.

In the example of "satisfying hunger", you don't have to make that a separate decision (e.g. "I am now going to satisfy hunger.") Instead, you just add your hunger value as a consideration to anything that would involve moving to or eating food. Therefore, as hunger increases (or food state decreases), those behaviors will gradually rise in score until such point as they become urgent. There is no separate state... just behaviors that respond directly to those changing input values.

I ... I think I get it. I think I understand!

You're right, I was overthinking it.

It sounded daunting to program that many "solutions" for each agent to consider for their actions. But even if my initial design doc might try to organize those into groups of actions, that's just because it's easier to plan the game that way. In the code, it's still just a long list of actions.

Culling the agent's mental model seemed so necessary before. But you're right that 100 characters each considering 100 actions is still only 10000 calculations. And even so, there's some obvious culling to do in terms of "too far away, don't bother", or "don't check this every damn second".

Thanks so much everyone. Really appreciate all of this.

A mechanism to adjust priorities of different actions is an escalation graph (can also be modal if-then logic) where the priority increases as the needed resource inventory decrease. Then the available tasks get sorted under one metric - can be a metric for this type of task (vs priorities for Immediate threats - which generally trump 'maintenance' by a magnitude.)

Calculating the priority may have additional factors (like very close proximity making an opportunistic grab viable)

Ive found that trying to develop a general metric to make these decisions is the hardest part of doing this programming (so you will need to be able to make adjustments while tuning the game, and a modal graph is fairly easy to adjust (versus an equation)

Most games have fewer than 20 types of action so this is not hard to do.

Caveman has an unusually large number of possible situations it responds to - that's why i use a hierarchical approach to decision making - divide and conquer. Although in the end, they all result in some combo of turning, moving, and attacking.

you can also use a 'less frequent check' type of optimization where the lower priorities usually culled out still get run once in a while (particularly if the opportunity search uses ALOT of processing resources)

One of the problems with AI is that even though you want to follow up on one (the last) decided task, that it is good to process the Current situation fairly frequently to have the behavior reactive to a suddenly changing environment (meaning lots of constant re-detecting/re-classifying/re-evaluating even when it results in no change). And as the game situation gets more complex that processing increases geometrically.