git history really big, consider making new repos for each semester

[SamLau95]

Looks like the .git folder is really big:

$ du -sh data8assets/.git                                                                                  
1.1G    data8assets/.git

This is probably because git keeps the history of adding / removing big datasets.

Students had trouble downloading their work since tar would zip the .git folder too. For now I've told them to run tar on the data8assets/materials folder instead (~150 MB).

Since git clone causes a full download of the history, next semester we'll have 500 students trying to download a >1 GB repo which seems wasteful.

We can avoid this by making new repos for each semester (materials_sp17, materials_fa17, etc), or by squashing the history for this repo. I prefer the first option since it avoids rewriting git history which makes git pull more complicated for the staff.

[yuvipanda]

We totally should do this. We could archive this repo, and then just have a new repo here that just has squashed history. Or just have a specific repo per semester as @SamLau95 mentioned. I think having a full checkout here for each student is going to cause problems with the fall infrastructure.

@SamLau95 @papajohn objections to doing this?

[SamLau95]

@yuvipanda Sounds like a great idea!

[papajohn]

@vinitra

…

On Mon, Aug 7, 2017 at 11:45 PM, Sam Lau ***@***.***> wrote: @yuvipanda <https://github.com/yuvipanda> Sounds like a great idea! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATikZxI155KmPvF1u3A4s9QqgRZKIK0ks5sWAQGgaJpZM4LQiJL> .

[vinitra-zz]

On it! @papajohn

[papajohn]

Fine with me. Let's make sure all the old stuff stays published and that all links are permanent.

…

On Mon, Aug 14, 2017 at 11:13 AM, Vinitra Swamy ***@***.***> wrote: On it! @papajohn <https://github.com/papajohn> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATikS9A_REnumpamcZOQjztL5Wo25SPks5sYI5JgaJpZM4LQiJL> .

[yuvipanda]

Thanks @vinitra. I see the new materials repo, and it's at 90M - which is much better! It looks like you deleted old files as a commit - data-8/materials-fa17@ae3fcd3. Since git keeps all history, deleting like this doesn't actually reduce the size of the repo! You should use something like https://help.github.com/articles/removing-sensitive-data-from-a-repository/. Let's do that before classes start - am happy to spend some time with you to get it done.

[vinitra-zz]

@yuvipanda thanks for the insight! I think I fixed it... can you confirm?

[yuvipanda]

Nope, it's still there: data-8/materials-fa17@51bbdfd

[yuvipanda]

@vinitra I've done some force pushing and cleaned it out. People who had clones locally might have to re-clone - let me know if people run into issues and I can help.