When can your right to assembly be limited/ taken away?

answer; taking up arms to defend the nation;

congress also has implied powers, which derive from the necessary and proper clause of the constitution and permit congress "to make all laws which shall be necessary and proper for carrying into execution the foregoing powers, and all other powers vested by this constitution in the government of the united states.