Theology 101: “My kingdom is not of this world” does not mean Christ’s kingdom is merely spiritual, heavenly, moral, or ethereal. It means, primarily, His kingdom is not “worldly”—it does not follow worldly principles, employ worldly tactics, or seek worldly ends. - Dr. Michael J. Svigel

The statement from John's recording of Jesus' words is often used to support a "spiritual only" kingdom. It's helpful to examine the context of the other "not of the world" statements of Jesus only a few hours before in John 17.
 
He said of His disciples that "they are still in the world", though they are "not of the world any more than I am of the world".
 
And let's not forget the testimony of the prophets regarding the physical aspects of the future kingdom of God on earth

"In the last days the mountain of the house of the LORD
     will be established as the chief of the mountains;
it will be raised above the hills,
     and all nations will stream to it."
- Isaiah

And Jesus own command that we should pray...

"Your kingdom come..."