Determined AI (master API, generic task kill/pause/unpause handlers): The generic task kill, pause and unpause

Impact

The generic task kill, pause and unpause endpoints run no ownership check, so any authenticated user terminates or suspends another user's tasks. On a shared GPU cluster that means one tenant can wipe out days of another team's training run and free their GPUs, with no permission escalation required.

Who can reach it

Any authenticated Determined user with network access to the master API. Affects versions up to and including 0.38.1.

What to do

Upgrade Determined past 0.38.1 and restart the master. Until then, check master audit logs for kill/pause calls whose caller does not match the task owner - the behaviour is indistinguishable from a legitimate request in the task's own history.

References