The man himself.
Having read so many books by a single author in such a short time, you can't help but notice certain patterns in his or her work, and Kay's work is no different, in that respect. But the patterns that he follows are themselves what sets him apart from other fantasy authors (I've read that he prefers not to be pigeonholed there, but it'll have to do for the moment).
The main idea running through almost all of his books is, of course, how his novels are based on historical events. His two most recent were inspired by key events in Chinese history, and previous ones have explored settings as diverse as the Byzantine Empire, Moorish Spain, and the Albigensian Crusade in southern France. The only ones that didn't hew to this historical fantasy mode were his opening trilogy, The Fionavar Tapestry, and a sort-of sequel called Ysabel.
Being kind of an obsessive completist, I started with the Fionavar books, several years ago, although I wouldn't say they're Kay's best. They felt a little too indebted to Tolkien - he helped edit some of Tolkien's papers while still at university - and ended up pulling in a few threads (no pun intended) that felt sort of out of place to me (think King Arthur and Lancelot). But for whatever reason, after finishing those books, Kay went off in a completely new direction, and I think it's served him well, allowing him to avoid the usual settings reminiscent of medieval northern Europe, and the usual plots of dark lords and sieges.
He's said himself that he's interested in the historical settings, but with fantasy trappings, because it frees him from having to write exactly what happened, or portray historical figures as they "really" were. That said, I've learned from hard experience to avoid reading the Wikipedia entries on those specific historical periods until finishing the book in question, otherwise I end up getting some brutal spoilers.
The other striking thing about his work is how he uses female characters. Some fantasy authors portray women really badly, alternately as scolds or sex toys; others arbitrarily make them badass soldiers, which is a neat idea in itself, but sometimes gets overused. Given that Kay's books are mostly based on historical events, his women rarely pick up swords, but even when their societies deny them power, they still have agency. A notable example is his 1992 book, A Song for Arbonne, based on the Albigensian Crusade, which took place in Provence; there he posits a society ruled by women, where the arts most appreciated at court are wit and poetry. Perhaps to contrast it with the prevailing themes of epic fantasy literature, the antagonists are from a heavily sexist, warlike country to the north, which despises Arbonne enough to eventually launch an invasion.
Arbonne was the first book of his that I read after the Fionavar books, and this struck me, in no small part because I'd found his treatment of one of his female characters in the first volume, The Summer Tree, to be pretty awful. Where most of the other main characters in the book - a group of Canadian students who travel to a magical fantasy world - gain powers of their own, one of the women sits around being pretty for the entire book, until she's captured by the evil god and raped. I remember almost putting the book down at the time, because it felt as if she'd been put in there exclusively for that (although, unlike, say, Paul Kearney, Kay doesn't lovingly describe it happening).
Now, what happens to the character in the next two books kind of refutes that, but not entirely. So I've always wondered if his amazing female characters since then - like the Empress Alixana or the physician Jehane - were intended to make up for that early mistake. Whatever his intentions, Kay's female characters set a standard that every fantasy author should aspire to.
Another notable thing about Kay's books is that, poetic and beautiful as they are, they also don't shy away from warfare and violence (or sex). Much like I said in my laudatory post about Joe Abercrombie's work, Kay allows you to see the violence that characterized these societies, and its effects as they ripple out from one act, seemingly insignificant, without flinching or even reveling in it. The description of what happens to a pair of characters toward the end of The Lions of Al-Rassan, set in an analogue to Moorish Spain, is brutal and heart-rending, but also rings true.
If I have any criticism of Kay's work, it's that his style has perhaps become more solidified - I won't say set in its ways: there were a few too many points in River of Stars, his latest, where the omniscient narrator talks about how a character's actions affect the rest of his or her life, or the history of their nation. It was also sometimes difficult to tell why certain characters did certain things; but for all that, the very end of the book was amazing, and like in all of his books, the prose does a fantastic job of conveying the surroundings to the reader.
And that, in the end, is why I believe that every would-be fantasy author should read Guy Gavriel Kay. His prose is amazing, probably the best in the business - the only one I think can come close (and that's because he's funny) is Joe Abercrombie. F Scott Fitzgerald said that a person who wants to write fiction should read six top-flight authors each year - Guy Gavriel Kay should comfortably sit in that rank of writers.