Add the following to <VirtualHost>,

# Deny access to git/svn files
<Directorymatch "^/(.*/)*\.(git|svn)/">
Order deny,allow
Deny from all
</Directorymatch>

According to this answer, a .htaccess file with this will do the same,

RewriteEngine On
RewriteRule "^(.*/)?\.(git|svn)/" - [F,L]