I am trying to implement a nested state machine with plutus and I'm wondering if it is even a possibility.
The answer is yes - qualified. It will eventually depend on what you have in mind.
The code that only runs off-chain is limited by the capabilities of Haskell.
But the part of the off-chain code that is executed on-chain (validator) is subject to the limitations of the Plutus Core language which is what runs on-chain. However, I don't know what they are in this case.