Multiple scenes loading ontop of eachother

Using PUN 1.75

This is an odd bug. In my game there are 3 scenes. The MainMenu, LobbyRoom, and Game. The game boots up in the Main Menu. When a user "Creates a Room" they make a Photon Room and load into the LobbyRoom scene. Once all players ready up it loads into the Game scene. In the Game scene all players can vote to "Restart". If they do they load back into the LobbyRoom scene.
The issue is that if the player(s) "Restart" and a new player joins the Photon Room (or a player leaves and re-joins) all of the players who were in the room before the new player joined "Freeze" (run very slowly) and get messages in their console that "QueueIncomingReliableWarning. This client buffers many incoming messages...".
And the new player (and only the new player) that joins sees both the LobbyRoom scene & Game scene loaded on top of each other. (The "QueueIncomingReliableWarning" might only be happening because we have quite a few objects that spawn in in the same position and are trying to re-orient themselves).

I don't actually see 2 scenes loaded in the "Hierarchy", but all of the objects that were loaded in the Game scene are loaded in.

My first guess was that it was remembering both scenes being loaded and syncing both scenes. But as I'm writing this I have a second guess. All of the objects that we load in we use PhotonNetwork.InstantiateSceneObject(). Is it possible that these objects are still in memory after we switch scenes? (but for some reason when we actually LoadLevel for the players in the room they are "destroyed")

We have this variable set in the MainMenu: PhotonNetwork.automaticallySyncScene = true;
And to transition between scenes only the MasterClient loads the next scene: PhotonNetwork.LoadLevel(SceneNameRoom);