Its because of this file: ShaderCache.cooked It resides in your C:\Users\Username\Documents\Elder Scrolls Online\live folder. I don't know why they chose to save shader info (mostly shadows from what I've read, not sure though) to a cache file. See, the game .exe is only 32 bit, which means that it can only access 4 gigs of RAM. So they try to use a "cheat", similar to what SWTOR uses, that allows the game to use more than 4g of RAM on systems that have excess. I suspect that file is part of their system, but who knows.
Bottom line is that you can do nothing about how long it takes to build/rebuild that file. Its annoying.
Latest versions of nVidia drivers also create a shader cache, but it doesn't affect the load times of my other games, at least not that I've been able to notice. You can at least toggle that one for On and Off, but I'd recommend leaving it at default setting.
Of note, folks might find this Reddit thread of interest: http://www.reddit.com/r/elderscrollsonline/comments/29px9g/simple_fps_issue_workaround_that_fixed_my_problem/