Fantasy authors build their own worlds and people them with humans and other unusual species and aspects to make the world unique. What readers and some authors don't realize is all stories partake in world building.
A romance writer for instance has the heroine and hero and around them is their world. A contemporary world peopled with friends and adversaries, but how they react and the influences of their world help to build not only who they are, but how their surroundings look to the reader. The author must build the scene they want to get the reactions of their characters and thus the story to play out in the way they anticipate.
Many authors will not agree they have to build a world, especially those who write in the pantser style (this means not plotting, just sit down and write with only a general idea of where the story is going). This means the story evolves as you go along rather than planning everything out before you start writing. I am a pantser writer and I find I do very little basic world building before I start and the rest evolves as I write. As I edit the drafts (usually up to three times before sending it to the publisher) the world pulls tighter together, including the influence of the characters on the scenes and people around them.
Without my characters there would be no world as they are the center of the story and the hub everything revolves around. From them comes the description and structure of the world and time they live in. Their supporting characters paint the backdrop. The description is not put down as "The land was green" etc. the image comes from the interaction of the characters with their surroundings and peers.
Bear in mind every book is like a play on a stage, created not only by the author, but also by the characters, supporting characters and their actions. Together they influence the story to bring the full essence of it to fruition. The stage is the world, the play brings the story to life.
Of course fantasy writers have more leeway, but sometimes this can be more challenging. When the imagination takes full reign, it has to be pulled in by reality. Everything has to merge together in a way the reader can understand and empathize with, plus they must fit into the world the author has created. Good luck with your building and I hope to see your world appear on my bookshelf sometime soon.