This is a problem I've been having for a few weeks now, since I started using git and the git-svn plug-in to operate on a Subversion repository. Today I figured out the solution.
When you git clone a repository which has been used with Subversion and git-svn, none of the extra metadata is cloned with it. That's expected, so you have to do git svn init again to configure git-svn in this new repository. However, this does not save you from git-svn then downloading everything from Subversion! This can easily take hours, if not days, on a large repository.
The trick is simple enough: git update-ref refs/remotes/git-svn master. Note that you might need to use a different branch name in place of "master", depending on your git repository's set up. All this does is tell git-svn that the latest Subversion commit is the latest git commit on branch "master".
Now when you do git svn fetch, the worst it'll do is "Rebuilding ...rev_map..." which should only take a minute or two and is entirely local.
$ git clone https://git.example.com/
normal clone stuff
$ git svn init https://svn.example.com/trunk
$ git update-ref refs/remotes/git-svn master
$ git svn fetch
Rebuilding .git/svn/refs/remotes/git-svn/.rev-map.guid ...
r1 = sha1
...
Done rebuilding .git/svn/refs/remotes/git-svn/.rev-map.guid
Success!