...or a squarish format. (As always, there are exceptions---for example shape books, which have die-cut, non-rectangular silhouettes.)
Those working with picture books usually talk about their trim size, which is the size of the page. The physical book is a bit larger, as the hard cover extends slightly beyond the page on all four sides. The minimum trim size in today's publishing world would be a 9 x 9 inch square, or an 8 x 10 rectangle. A publisher probably would not produce a picture book smaller than this, for fear that it would get lost amidst the competition---literally covered up by larger books in the bookstore display, for example.
But in spite of today's philosophy of "Bigger is Better", there is an upper limit to the size of picture books. We need to be able to physically hold them and read them, aloud or to ourselves, with ease and comfort. The books need to fit onto the shelves of libraries and households. And over-sized books are more costly for the publisher to produce. So the maximum trim size generally would be an 11 x 11 inch square, or a rectangle with a maximum 11 inches on the longest side, such as 9 x 11 or 10 x 11. (When I was researching all this recently at my local independent bookstore CHANGING HANDS (http://www.changinghands.com/) I saw one exception to this upper limit: a book with a 10 1/2 x 12 horizontal format---and a celebrity name on its cover.)
Within these parameters, an illustrator would seemingly have unlimited choices---for example, she/he could ask for a trim size of 9 1/4 x 10 7/8. But the final decision will actually be made by the publisher, though hopefully in consultation with the illustrator. Since the book will be printed on very large sheets of paper (with one complete printed picture book fitted onto one sheet), and since various papers come in various sheet sizes, the sheet size of the paper will therefore determine the trim size of the book. And the choice of the paper itself will most likely be made by the publisher, based on the paper's characteristics, and its cost.