Add git-lfs to image

[jameshearttech]

I'm requesting git-lfs be added to the image. This issue came up back in 2019 #174. The issue was closed without adding git-lfs to the image. I tested v3.6.2 by adding git-lfs to the image and I'm syncing LFS files from BitBucket without any additional configuration.

This is my dockerfile.

FROM k8s.gcr.io/git-sync/git-sync:v3.6.2
USER root
RUN apt-get update
RUN apt-get -y install git-lfs
USER 65533:65533

This is my container spec within my deployment in k8s.

spec:
  template:
    spec:
      containers:
        - name: git-sync-lfs
          image: example-docker-hub/git-sync-lfs:v3.6.2
          args:
            - -ssh
            - -repo=git@bitbucket.org:example-private-repo/lfs.git
            - -root=/data
            - -dest=lfs
            - -branch=master
            - -depth=1
            - -wait=60
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: lfs-data
              mountPath: /data
            - name: git-secret
              mountPath: /etc/git-secret
          securityContext:
            runAsUser: 65533

[thockin]

I would want an e2e test for this - how do I force this to be used so I know it works?

[jameshearttech]

Do existing tests not cover this? Can I not just add git-lfs package in a Dockerfile using apt-get then run tests on new image?

My background is sysadmin. I'm new to devops. I know some bash, docker, and k8s. You want me to write a test to prove that it works? Not sure how to do that.

I forked the repo. I'm looking at Dockerfile.in and test_e2e.sh. I'm not sure where to add git-lfs package in Dockerfile.in. Do I pass it an image:tag as an argument to run tests?

[thockin]

If you can show me how to craft an e2e setup which specifically proves LFS works, that would be great. You can see in that file that each testcase is a little snip of shell code which sets up a scenario. What I don't quiote grok is how to set up a repo to use LFS and how to *prove* that it works.

…

On Thu, Dec 22, 2022 at 4:56 PM James Ringer ***@***.***> wrote: Do existing tests not cover this? My background is sysadmin. I'm new to devops. I know some bash. You want me to write a test to prove that it works? Not sure how to do that. I went ahead and forked the repo. I'm looking at test_e2e.sh. How can I help? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[jameshearttech]

Does this help? GitHub - Configuring Git LFS

GitHub provides some storage and bandwidth for LFS for free. GitHub - About Storage and Bandwidth

We can do a call and work through it together if you'd like. I set it up on BitBucket, but it's probably similar on GitHub. The BitBucket repo I have setup is private, but I could setup a public repo if that helps.

Afaik the only thing that will be different after adding the git-lfs package will be that if a repo contains lfs files they will be synced rather than the text files that point to them. This makes me think existing tests should be fine, but I'm far from and expert.

[thockin]

To do an e2e I need some way to replicate it locally or at least without any auth to an external service. Possible?

…

On Thu, Dec 22, 2022, 5:23 PM James Ringer ***@***.***> wrote: Does this help? GitHub - Configuring Git LFS <https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage> We can do a call and work through it together if that helps. I set it up on BitBucket, but it's probably similar on GitHub. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVG5E3GCH32QE7KMKMDWOT5I7ANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

[jameshearttech]

If you setup a public GitHub repo with LFS I don't think you need authentication.

[jameshearttech]

Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?

[thockin]

The existing e2e tests use a `file://` repo or an SSH repo (still backed by files). It looks like we can build an LFS test-server binary ( https://github.com/git-lfs/lfs-test-server) to run tests against, but there's not a standard apt pkg for it, so we'll have to pull and build it as part of the test. Look in _test_tools. That's not impossible, it just makes this a little more involved. At least it is written in Go, so we can drop it into tools/go.mod. :) If you have bandwidth, it's appreciated, otherwise I'll get to this eventually, I hope. :)

…

On Fri, Dec 23, 2022 at 7:10 AM James Ringer ***@***.***> wrote: Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo? — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVFGCCNKCP5ONWDXENDWOW6E7ANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

[jameshearttech]

In my fork I have setup lfs-test-server using the sshd folder as an example.

_test_tools
├── lfs-test-server
│   ├── Dockerfile
│   ├── README.md
│   └── run.sh
├── ncsvr
│   ├── Dockerfile
│   ├── README.md
│   └── ncsvr.sh
└── sshd
    ├── Dockerfile
    ├── README.md
    ├── sshd.sh
    └── sshd_config

I am using env vars LFS_USERNAME and LFS_PASSWORD to configure lfs-test-server management username and password. Looking at sshd example I'm assuming we don't want these committed to the repository. Not sure how I'm going to do that. If you don't set these the management interface is not accessible. Also, it looks like you have to login and create a user to use lfs-test-server. Not how I'm going to work around that either. Any ideas?

[thockin]

Ugh, what a cumbersome piece of software to automate. How about something like this Dockerfile:

FROM alpine AS base                                                                                                                                   

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
    alpine-baselayout \
    apk-tools \
    curl \
    && true

###############

FROM scratch

ENTRYPOINT []
WORKDIR /

COPY --from=base /out/ /

COPY lfs-test-server /

RUN LFS_ADMINUSER=admin \
    LFS_ADMINPASS=admin \
    timeout 2 \
    /lfs-test-server & \
    sleep 0.5 \
    && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e

ENTRYPOINT ["/lfs-test-server"]

I did a quick test and that SEEMS to work, as long as lfs-test-server is built with CGO_ENABLED=0 (static binary).

Now you can run the image in docker, get the IP (see ssh examples) and us username "e2e" password "e2e".

[jameshearttech]

I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.

[thockin]

That's because it's not easy to make "trivial" SSH keys, whereas this case is super easy :)

…

On Mon, Dec 26, 2022 at 4:34 PM James Ringer ***@***.***> wrote: I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed. — Reply to this email directly, view it on GitHub <#654 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVFVFQ4LEKCP6F7JCZLWPI2TDANCNFSM6AAAAAATHFYSJ4> . You are receiving this because you commented.Message ID: ***@***.***>

[jameshearttech]

That makes sense. I just wanted to check before moving forward. Btw adding the user with curl that way is just what I was looking for. I'll keep working on this. Thanks.

[jameshearttech]

Haven't circled back to this yet, but I will. Please don't let the issue go stale and close. Thanks!

[thockin]

Hi. I am hoping to have a v4.0.0 release candidate some time in the next month or so - would love to have this in. No pressure, we can always add it later.