It's spelled “Puzzleburg” with a “u” in “burg” because that's the more common spelling for cities across the world.  For example, the cities of Hamburg (Germany), Johannesburg (South Africa), Salzburg (Austria), St. Petersburg (Russia and FL, USA), Gettysburg (PA, USA), and Williamsburg (NY and VA, USA) all spell it this way.  Some prominent cities spell it with an extra “h” at the end, such as Edinburgh (Scotland) and Pittsburgh (USA), but few cities known worldwide spell it with an “e.” There's the country of Luxembourg containing the city of Luxembourg (aka Luxembourg City), which throws an extra "o" in there, but let's not get started on that.

Yes, I did in fact research the spelling of various cities in order to settle on how exactly to spell "Puzzleburg."  If that seems like an excessive amount of attention to detail, well, I'll just say the attention to detail doesn't stop there.

Bonus question: Fine, you decided exactly how you wanted to spell "Puzzleburg," but why is this town named that at all?  What's the in-game explanation?  Where's the attention to detail on that, huh?

As you'll find out as you play the game, the town's mayor is named Adam Puzzleburg, and he comes from a long line of Puzzleburgs who were mayor before him.  So it's possible that the town is named after the first mayor, whose last name was Puzzleburg.  Or perhaps the mayor changed his name to match the name of the town, who's to say?

Regardless, you should at least note what shape the island is. As you can see from one of the screenshots on the company home page here (and on the Steam page here), the island the town is located on is shaped like a puzzle piece, so perhaps that's where the town's name originally comes from.  You wanted attention to detail, so there you go.