Hi, I want clone a gitlab repo with id and token during job execution. To hide the token I am passing the value in my job. How can I export the value of GIT_TOKEN in environment file of LAVA? So that whenever job is executed it will read the value from its environment file. I tried to use export and tried to add "GIT_TOKEN=abcd" in env.yaml file. But I guess its not correct format. I couldn't find what kind of values we can add in env.yaml in documents. I also added GIT_TOKEN value in /etc/profile of the LAVA server still the job didn't pick it. Could you please suggest how can we clone the repo in LAVA job without exposing its password. Also I understand that ssh key of root will work but I want to avoid using ssh key of root. Kindly suggest - test: timeout: minutes: 15 definitions: - repository: https://gitid:%24GIT_TOKEN@gitlab.com/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: pipeline name: cis-benchmark
Sweta,
On Thu, Aug 29, 2024 at 6:19 AM sweta.ghosh@nagarro.com wrote:
Hi, I want clone a gitlab repo with id and token during job execution. To hide the token I am passing the value in my job. How can I export the value of GIT_TOKEN in environment file of LAVA? So that whenever job is executed it will read the value from its environment file. I tried to use export and tried to add "GIT_TOKEN=abcd" in env.yaml file. But I guess its not correct format. I couldn't find what kind of values we can add in env.yaml in documents. I also added GIT_TOKEN value in /etc/profile of the LAVA server still the job didn't pick it. Could you please suggest how can we clone the repo in LAVA job without exposing its password. Also I understand that ssh key of root will work but I want to avoid using ssh key of root. Kindly suggest
Jobs are executed on the dispatcher. Unless your server and dispatcher run on the same machine, setting environment on server won't do the trick.
- test: timeout: minutes: 15 definitions:
- repository: https://gitid:%24GIT_TOKEN@gitlab.com/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: pipeline name: cis-benchmark
The docs say that you can override some env variables of the job using env.yaml file: https://validation.linaro.org/static/docs/v2/proxy.html There are 2 options: - /etc/lava-server/env.yaml - for all workers - /etc/lava-server/dispatcher.d/<name>/env.yaml - for selected worker only In this case it talks about HTTP_PROXY, but I believe you can set GIT_TOKEN there. I don't know if it's going to work. Best to try and report back :)
Your only other option is limiting the job visibility to "personal" or "group". This way the job data won't be available to the world and you can use the token in the definition. It's not an ideal solution, but I don't think there is a better one available.
Best Regards, Milosz
lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
Hello Milosz,
Thanks for replying.
I have a single node setup, so worker and server is same. I updated /etc/lava-server/env.yaml file with GIT_TOKEN value like below. But I get error Invalid yaml when I execute job: root@debian:~# cat /etc/lava-server/env.yaml # Configure the environment that will be used within the server and the dispatcher.
# If set to true, all variables inherited by the process will be dropped. # The process environment is then fully controlled but you should then list all # variables that your process needs. purge: true
# List the variables that will be removed from the inherited environment. # If purge is 'true', this list is useless. # default: empty list #removes: #- HTTP_PROXY #- HTTPS_PROXY #- FTP_PROXY #- NO_PROXY
# A dictionary of (key, value) that will be added to the inherited environment. # If a key does not already exist in the inherited environment, it's added. # default: an empty dictionary overrides: LC_ALL: C.UTF-8 LANG: C # http_proxy: http://lava-lab-proxy # https_proxy: http://lava-lab-proxy # ftp_proxy: http://lava-lab-proxy PATH: /usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin GIT_TOKEN=abcd
root@debian:~# systemctl status lava-server-gunicorn.service ● lava-server-gunicorn.service - LAVA server gunicorn Loaded: loaded (/lib/systemd/system/lava-server-gunicorn.service; enabled; preset: enabled) Active: active (running) since Thu 2024-08-29 13:56:56 IST; 1min 9s ago Main PID: 15978 (gunicorn: maste) Tasks: 9 (limit: 9429) Memory: 298.8M CPU: 5.375s CGroup: /system.slice/lava-server-gunicorn.service ├─15978 "gunicorn: master [lava_server.wsgi]" ├─15980 "gunicorn: worker [lava_server.wsgi]" ├─15981 "gunicorn: worker [lava_server.wsgi]" ├─15982 "gunicorn: worker [lava_server.wsgi]" └─15983 "gunicorn: worker [lava_server.wsgi]"
Aug 29 13:58:05 debian gunicorn3[15983]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Aug 29 13:58:05 debian gunicorn3[15983]: File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view Aug 29 13:58:05 debian gunicorn3[15983]: return view_func(*args, **kwargs) Aug 29 13:58:05 debian gunicorn3[15983]: ^^^^^^^^^^^^^^^^^^^^^^^^^^ Aug 29 13:58:05 debian gunicorn3[15983]: File "/usr/lib/python3/dist-packages/lava_scheduler_app/views.py", line 1201, in internal_v1_jobs Aug 29 13:58:05 debian gunicorn3[15983]: env_str = config("env") Aug 29 13:58:05 debian gunicorn3[15983]: ^^^^^^^^^^^^^ Aug 29 13:58:05 debian gunicorn3[15983]: File "/usr/lib/python3/dist-packages/lava_scheduler_app/views.py", line 1197, in config Aug 29 13:58:05 debian gunicorn3[15983]: raise OSError( Aug 29 13:58:05 debian gunicorn3[15983]: OSError: [Errno ] Invalid YAML file for debian: env file
Thanks, Sweta ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Thursday, August 29, 2024 1:53 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Not able to set variable
[Email from a non-Nagarro source: please exercise caution with links and attachments]
Sweta,
On Thu, Aug 29, 2024 at 6:19 AM sweta.ghosh@nagarro.com wrote:
Hi, I want clone a gitlab repo with id and token during job execution. To hide the token I am passing the value in my job. How can I export the value of GIT_TOKEN in environment file of LAVA? So that whenever job is executed it will read the value from its environment file. I tried to use export and tried to add "GIT_TOKEN=abcd" in env.yaml file. But I guess its not correct format. I couldn't find what kind of values we can add in env.yaml in documents. I also added GIT_TOKEN value in /etc/profile of the LAVA server still the job didn't pick it. Could you please suggest how can we clone the repo in LAVA job without exposing its password. Also I understand that ssh key of root will work but I want to avoid using ssh key of root. Kindly suggest
Jobs are executed on the dispatcher. Unless your server and dispatcher run on the same machine, setting environment on server won't do the trick.
- test: timeout: minutes: 15 definitions:
- repository: https://gitid:%24GIT_TOKEN@gitlab.com/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: pipeline name: cis-benchmark
The docs say that you can override some env variables of the job using env.yaml file: https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvalidation...https://validation.linaro.org/static/docs/v2/proxy.html There are 2 options: - /etc/lava-server/env.yaml - for all workers - /etc/lava-server/dispatcher.d/<name>/env.yaml - for selected worker only In this case it talks about HTTP_PROXY, but I believe you can set GIT_TOKEN there. I don't know if it's going to work. Best to try and report back :)
Your only other option is limiting the job visibility to "personal" or "group". This way the job data won't be available to the world and you can use the token in the definition. It's not an ideal solution, but I don't think there is a better one available.
Best Regards, Milosz
lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
On Thu, Aug 29, 2024 at 9:31 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Hello Milosz,
Thanks for replying.
I have a single node setup, so worker and server is same. I updated /etc/lava-server/env.yaml file with GIT_TOKEN value like below. But I get error Invalid yaml when I execute job: root@debian:~# cat /etc/lava-server/env.yaml # Configure the environment that will be used within the server and the dispatcher.
# If set to true, all variables inherited by the process will be dropped. # The process environment is then fully controlled but you should then list all # variables that your process needs. purge: true
# List the variables that will be removed from the inherited environment. # If purge is 'true', this list is useless. # default: empty list #removes: #- HTTP_PROXY #- HTTPS_PROXY #- FTP_PROXY #- NO_PROXY
# A dictionary of (key, value) that will be added to the inherited environment. # If a key does not already exist in the inherited environment, it's added. # default: an empty dictionary overrides: LC_ALL: C.UTF-8 LANG: C # http_proxy: http://lava-lab-proxy # https_proxy: http://lava-lab-proxy # ftp_proxy: http://lava-lab-proxy PATH: /usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin GIT_TOKEN=abcd
It needs to be a valid YAML. You need to align the indentation with other variables in "overrides" section. Also use GIT_TOKEN: "<token value>" instead of "="
Best Regards, Milosz
Thanks Milosz, I am not getting error anymore. But my job is still not able to take the value. Please find the test action I tried : - test: timeout: minutes: 5 definitions: - repository: https://lava_test_token:%24%7BGIT_TOKEN%7D@gitlab.com/testing/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: lava-pipeline name: cis-benchmark
Is there way to call the value in job like in context or environment?
Thanks, Sweta
________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Thursday, August 29, 2024 2:04 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Not able to set variable
[Email from a non-Nagarro source: please exercise caution with links and attachments]
On Thu, Aug 29, 2024 at 9:31 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Hello Milosz,
Thanks for replying.
I have a single node setup, so worker and server is same. I updated /etc/lava-server/env.yaml file with GIT_TOKEN value like below. But I get error Invalid yaml when I execute job: root@debian:~# cat /etc/lava-server/env.yaml # Configure the environment that will be used within the server and the dispatcher.
# If set to true, all variables inherited by the process will be dropped. # The process environment is then fully controlled but you should then list all # variables that your process needs. purge: true
# List the variables that will be removed from the inherited environment. # If purge is 'true', this list is useless. # default: empty list #removes: #- HTTP_PROXY #- HTTPS_PROXY #- FTP_PROXY #- NO_PROXY
# A dictionary of (key, value) that will be added to the inherited environment. # If a key does not already exist in the inherited environment, it's added. # default: an empty dictionary overrides: LC_ALL: C.UTF-8 LANG: C # http_proxy: http://lava-lab-proxy # https_proxy: http://lava-lab-proxy # ftp_proxy: http://lava-lab-proxy PATH: /usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin GIT_TOKEN=abcd
It needs to be a valid YAML. You need to align the indentation with other variables in "overrides" section. Also use GIT_TOKEN: "<token value>" instead of "="
Best Regards, Milosz
On Thu, Aug 29, 2024 at 10:29 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Thanks Milosz, I am not getting error anymore. But my job is still not able to take the value. Please find the test action I tried :
- test: timeout: minutes: 5 definitions:
- repository: https://lava_test_token:%24%7BGIT_TOKEN%7D@gitlab.com/testing/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: lava-pipeline name: cis-benchmark
Is there way to call the value in job like in context or environment?
I hoped this line did the trick: https://gitlab.com/lava/lava/-/blob/master/lava_dispatcher/utils/vcs.py?ref_...
In your job log you should see the line: Running '/usr/bin/git clone --depth=1 http://github.com/linaro/test-definitions.git /var/lib/lava/dispatcher/tmp/50453/lava-overlay-eysulgb5/home/ Obviously the URL and path will be different, but this should print the command _before_ variable substitution. You can experiment with vcs.py and move the debug line below the substitution. This way you will know whether the environment was passed properly to the GitHelper.
I don't know any other tricks, sorry.
Best Regards, Milosz
Thanks, Sweta
From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Thursday, August 29, 2024 2:04 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Not able to set variable
[Email from a non-Nagarro source: please exercise caution with links and attachments]
On Thu, Aug 29, 2024 at 9:31 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Hello Milosz,
Thanks for replying.
I have a single node setup, so worker and server is same. I updated /etc/lava-server/env.yaml file with GIT_TOKEN value like below. But I get error Invalid yaml when I execute job: root@debian:~# cat /etc/lava-server/env.yaml # Configure the environment that will be used within the server and the dispatcher.
# If set to true, all variables inherited by the process will be dropped. # The process environment is then fully controlled but you should then list all # variables that your process needs. purge: true
# List the variables that will be removed from the inherited environment. # If purge is 'true', this list is useless. # default: empty list #removes: #- HTTP_PROXY #- HTTPS_PROXY #- FTP_PROXY #- NO_PROXY
# A dictionary of (key, value) that will be added to the inherited environment. # If a key does not already exist in the inherited environment, it's added. # default: an empty dictionary overrides: LC_ALL: C.UTF-8 LANG: C # http_proxy: http://lava-lab-proxy # https_proxy: http://lava-lab-proxy # ftp_proxy: http://lava-lab-proxy PATH: /usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin GIT_TOKEN=abcd
It needs to be a valid YAML. You need to align the indentation with other variables in "overrides" section. Also use GIT_TOKEN: "<token value>" instead of "="
Best Regards, Milosz
Thanks a lot Milosz, I was able to troubleshoot the issue. The value was picking up extra space, due to which I was getting the error. All in all the updating the env.yaml with variable helps. ________________________________ From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Thursday, August 29, 2024 3:09 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Not able to set variable
[Email from a non-Nagarro source: please exercise caution with links and attachments]
On Thu, Aug 29, 2024 at 10:29 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Thanks Milosz, I am not getting error anymore. But my job is still not able to take the value. Please find the test action I tried :
- test: timeout: minutes: 5 definitions:
- repository: https://lava_test_token:%24%7BGIT_TOKEN%7D@gitlab.com/testing/lava-tests.githttps://lava_test_token:$%7BGIT_TOKEN%7D@gitlab.com/testing/lava-tests.git from: git path: tests/cisscan/cis.yaml branch: lava-pipeline name: cis-benchmark
Is there way to call the value in job like in context or environment?
I hoped this line did the trick: https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com...https://gitlab.com/lava/lava/-/blob/master/lava_dispatcher/utils/vcs.py?ref_type=heads#L57
In your job log you should see the line: Running '/usr/bin/git clone --depth=1 https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%...http://github.com/linaro/test-definitions.git /var/lib/lava/dispatcher/tmp/50453/lava-overlay-eysulgb5/home/ Obviously the URL and path will be different, but this should print the command _before_ variable substitution. You can experiment with vcs.py and move the debug line below the substitution. This way you will know whether the environment was passed properly to the GitHelper.
I don't know any other tricks, sorry.
Best Regards, Milosz
Thanks, Sweta
From: Milosz Wasilewski milosz.wasilewski@foundries.io Sent: Thursday, August 29, 2024 2:04 PM To: Sweta Ghosh sweta.ghosh@nagarro.com Cc: lava-users@lists.lavasoftware.org lava-users@lists.lavasoftware.org Subject: Re: [lava-users] Not able to set variable
[Email from a non-Nagarro source: please exercise caution with links and attachments]
On Thu, Aug 29, 2024 at 9:31 AM Sweta Ghosh sweta.ghosh@nagarro.com wrote:
Hello Milosz,
Thanks for replying.
I have a single node setup, so worker and server is same. I updated /etc/lava-server/env.yaml file with GIT_TOKEN value like below. But I get error Invalid yaml when I execute job: root@debian:~# cat /etc/lava-server/env.yaml # Configure the environment that will be used within the server and the dispatcher.
# If set to true, all variables inherited by the process will be dropped. # The process environment is then fully controlled but you should then list all # variables that your process needs. purge: true
# List the variables that will be removed from the inherited environment. # If purge is 'true', this list is useless. # default: empty list #removes: #- HTTP_PROXY #- HTTPS_PROXY #- FTP_PROXY #- NO_PROXY
# A dictionary of (key, value) that will be added to the inherited environment. # If a key does not already exist in the inherited environment, it's added. # default: an empty dictionary overrides: LC_ALL: C.UTF-8 LANG: C # http_proxy: http://lava-lab-proxy # https_proxy: http://lava-lab-proxy # ftp_proxy: http://lava-lab-proxy PATH: /usr/local/bin:/usr/local/sbin:/bin:/usr/bin:/usr/sbin:/sbin GIT_TOKEN=abcd
It needs to be a valid YAML. You need to align the indentation with other variables in "overrides" section. Also use GIT_TOKEN: "<token value>" instead of "="
Best Regards, Milosz
lava-users@lists.lavasoftware.org
-
Milosz Wasilewski -
Sweta Ghosh -
sweta.ghosh＠nagarro.com