Re: [Lava-users] Git duplication in test jobs

On 14 May 2018 at 15:54, Denis HUMEAU denis.humeau@st.com wrote:

Hello Lava users,

I’m coming back with an old question that I think is worth asking again.

Here is an example of a test definition from Lava documentation:

test:

failure_retry: 3

timeout:

 minutes: 10

name: kvm-basic-singlenode

definitions:

   - repository: git://git.linaro.org/lava-team/lava-functional-tests.git

     from: git

     path: lava-test-shell/smoke-tests-basic.yaml

     name: smoke-tests

   - repository: http://git.linaro.org/lava-team/lava-functional-tests.git

     from: git

     path: lava-test-shell/single-node/singlenode03.yaml

     name: singlenode-advanced

     revision: 441b61

In this example, 2 tests are defined, and they’re contained in the same git repo. The test overlay will be built by Lava using this type of tree:

Lava_session_number/test1/

Lava_session_number/test2/

In each test folder, a git clone will be performed and data will be duplicated.

Nowadays, I understand that the trend is to test more in and embedded configurations, and less in nfs configurations. This is what we do for our product validation.

Embedded storages are limited, therefore we need to carefully think about our git management:

•   First strategy: have a single git containing all the tests. In
  

this case size of test overlay may increase dangerously

•   Second strategy: one git per test. This is not the spirit of a
  

git, but you solve the size issue.

The last time we asked the question, more than a year ago, there was no solution or workaround to prevent duplicating the same git in a Lava job. Has the situation changed?

Yes. shallow clones are the default. The .git directory can also be removed using the history option.

https://staging.validation.linaro.org/static/docs/v2/test-repositories.html

Symlinks from one repo to another didn't work, there were secondary problems causing test definitions to run more than once.

https://projects.linaro.org/browse/LAVA-1167

If not, I believe it would be great to have a mechanism to store gits no longer in the test folder itself, but in a common repository folder. I know this cannot be done in every case, for example you may have 2 tests on the same git but on a different branch, but this may be an optional feature trigger by a “shared” tag in the job definition.

Work is being planned for a revamp of the Lava Test Shell Definition to create a 2.0 schema. The 1.0 structure was preserved from V1 days to limit the scale of the migration to V2 and, as such, has inherited many problems from the V1 days.

If shared is declared, links to the common repo would be created in the test overlay folder.

For example:

test:

failure_retry: 3

timeout:

 minutes: 10

name: kvm-basic-singlenode

definitions:

   - repository: git://git.linaro.org/lava-team/lava-functional-tests.git

     from: git

      *shared*: yes

     path: lava-test-shell/smoke-tests-basic.yaml

     name: smoke-tests

   - repository: http://git.linaro.org/lava-team/lava-functional-tests.git

     from: git

      *shared*: yes

     path: lava-test-shell/single-node/singlenode03.yaml

     name: singlenode-advanced

What do you think? Am I the only one needing this, are you aware of a better solution?

That approach has known issues within the Lava Test Shell 1.0 support.

It will be some time before the 2.0 test shell is available - we are still in the early planning stage.