It wouldn't be the end of the world but it would not be ideal for me. The type of games that I like aren't available on the Switch and Fortnite is straight up trash on the Switch so that would not be enjoyable.
Personally, I have only a PC and play only PC video games. I'm not into consoles as 99% of the games I play are only for computers and having only the Nintendo Switch would be a nightmare for me. Well, it might be. I'm pretty sure there are cool games for it but I can't betray the PC gaming.
I think I'd rather keep the android phone. Most of the android phone are worth using as gaming device too. With 4GB RAM it's worth using android phone instead.