To quickly answer: it's not quite possible.
The long answer: With AudienceStream acting as a real-time action engine, it typically takes a user generated action to update their visitor profile. Omnichannel and the Tealium Collect API are two other ways to update a user based on an offline action, but you need to know the Visitor ID of your user in order to update them. However, it's common for a user to be part of an audience and not have a visitor identifer declared as part of their visitor profile. That said, hopefully you see the conundrum this puts you in when you want to remove all users from an audience.
Therefore, it might be easier to just create a brand new audience with the same criteria. This will effectively do what you need without a lot of leg work.
If you're purchased AudienceDB and your audience criteria require the visitor have a visitor ID assigned, you can always export the list of visitors to a CSV file and perform appropriate logic to remove them from an audience. Just make sure your audience criteria has been setup so that it's possible for the vistitor to be removed from the audience. For example, if the audience criteria is that the visitor's 90-day rolling order value be greater than 100, then set that value to 0.
Please let me know if this answer is satisfactory.