Exec Library

Introduction

A builder for executing commands.

Sources

https://github.com/milaboratory/platforma/tree/main/sdk/workflow-tengo/src/exec

API

Import

exec := import("@platforma-sdk/workflow-tengo:exec")

builder

Creates a new builder for constructing and executing commands.

Example

mixcrSw := assets.importSoftware("@platforma-open/milaboratories.software-mixcr:main")
mixcrForBackCmd := exec.builder().
  software(mixcrSw).
  secret("MI_LICENSE", "MI_LICENSE").
  printErrStreamToStdout().
  arg("presetSpecificationsForBack").
  addFile("preset.yaml", aFile).
  arg("preset.yaml").
  arg("presetForBack.json").
  saveFileContent("presetForBack.json").
  run()
  
presetSpecForBackContent := mixcrForBackCmd.getFileContent("presetForBack.json")

Builder API

cmd

Sets the command to be executed, e.g. /usr/bin/env or sh.

Arguments:

• commandName: string - the name of the command.

Example:

exec.builder().cmd("/usr/bin/bash")

software

Sets the software to be executed.

Arguments:

• software: softwareInfo - the software to execute that can be got from assets library.

Example:

mixcrSw := assets.importSoftware("@platforma-open/milaboratories.software-mixcr:main")
cmd := exec.builder().software(mixcrSw)

arg

Adds an argument to the command.

Arguments:

• arg: string - the argument to add.

Example:

exec.builder().
  cmd("aCommand").
  arg("argument1").
  arg("argument2")

env

Sets an environment variable.

Arguments:

• name: string - the name of the environment variable.
• value: string - the value of the environment variable.

Example:

exec.builder.
  cmd("/bin/bash").
  arg("-c").
  arg("echo $ANSWER_TO_THE_ULTIMATE_QUESTION").
  env("ANSWER_TO_THE_ULTIMATE_QUESTION", "42")

envMap

Sets multiple environment variables.

Arguments:

• envMap: map[string]string - a map of environment variables.

Example:

exec.builder.
  cmd("/bin/bash").
  arg("-c").
  arg("echo $FOO$BAR").
  envMap({"FOO": "4", "BAR": "2"})

secret

Set environment variable to value of given secret.

Arguments:

• secretName: string - name of secret to pass as environment variable.
• envName: string - name of environment variable to set with value of secret.

Example:

mixcrSw := assets.importSoftware("@platforma-open/milaboratories.software-mixcr:main")
mixcrForBackCmd := exec.builder().
  software(mixcrSw).
  secret("MI_LICENSE", "MI_LICENSE")

inHeavyQueue

Execute command in the 'heavy' queue.

inMediumQueue

Execute command in the 'medium' queue.

inLightQueue

Execute command in the 'light' queue.

inUiQueue

Execute command in the 'ui-tasks' queue.

cache

Sets the cache time.

Arguments:

• time: duration - the cache time from times library.

nErrorLines

Sets the number of error lines to be captured.

Arguments:

• lines: number - the number of error lines.

addFile

Adds a file to the working directory

Arguments:

• fileName: string - the name of the input file.
• file: reference - a resource id of the file or a field that points to it.

Example:

exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("cat input.txt").
  addFile("input.txt", aFile)

addFiles

Adds multiple files to the working directory.

Arguments

• filesMap: map[string]reference - a map of file names to files' references.

Example:

exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("cat input1.txt; cat input2.txt").
  addFiles({
    "input1.txt": aFile1,
    "input2.txt": aFile2
  })

writeFile

Writes a file with a given content.

Arguments:

• fileName: string - the name of the content.
• data: either bytes or a reference to a value resource - an encoded primitive tengo value (map or array) or a reference to a resource from which data we create a file.

Example:

json := import("json")

exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("cat input.txt").
  writeFile("input.txt", json.encode(42))

mkDir

Creates a directory, it can be deeply nested. A path separator should be / even on Windows, path library will canonize it under the hood.

Arguments:

• dir: string - the directory to create.

Example:

exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("cat ./deeply/nested/dir/input.txt").
  mkDir("deeply/nested/dir").
  writeFile("input.txt", json.encode(42))

addAsset

Adds files from given asset into working directory.

Arguments:

• asset: assetInfo - asset info loaded from assets.importAsset().
• destinationDir: string - path inside working directory for files to be extracted from archive.
• pathsInArchive: string[] - paths inside the archive to be extracted into destinationDir. Empty array means 'unpack full archive'.

Example:

asset := assets.importAsset("@platforma-open/milaboratories.software-small-binaries:small-asset")

// here we unpack file2.txt into a root of the working directory.
run := exec.builder().
  addAsset(asset, ".", ["file2.txt"]).
  cmd("/usr/bin/env").
  arg("cat").
  arg("file2.txt").
  saveStdoutContent().
  run()

saveFile

Saves a file as a resource. One can use result.getFile function to get the resource.

Arguments:

• fileName: string - the name of the output file.

Example:

result := exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("echo 42 > output.txt").
  saveFile("output.txt").
  run()

aFile := result.getFile("output.txt")

saveFileContent

Saves a file as a content. One can use result.getFileContent function to get the resource's content.

Arguments:

• fileName: string - the name of the output value.

Example:

result := exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("echo 42 > output.txt").
  saveFileContent("output.txt").
  run()

number42 := result.getFileContent("output.txt")

saveFileSet

Saves files from the working directory using regular expressions. One can use result.getFileSet function to get the resource's content.

Arguments:

• name: string - a name to refer this file set in the outputs
• regex: string - regex pattern. One can use https://regex101.com/ to test its regular expressions (Golang flavor should be used).

Example:

result := exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("echo 42 > output.txt").
  saveFileSet("allFilesStartedWithLetterO", "^o.*").
  run()
  
files := result.getFileSet("allFilesStartedWithLetterO")

saveFileSetContent

Saves files content from the exec working directory using regex. One can use result.getFileSetContent to get all contents of all files.

Arguments:

• name: string - a name to refer this file set in the outputs.
• regex: string - regex pattern. One can use https://regex101.com/ to test its regular expressions (Golang flavor should be used).

Example:

result := exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("echo 42 > output.txt").
  saveFileSetContent("allFilesStartedWithLetterO", "^o.*").
  run()

files := result.getFileSetContent("allFilesStartedWithLetterO")

processWorkdir

Run a specified template after a command execution.

Arguments:

• name: string - the name of the processor. After the builder ran, results of the processor can be got by its name via execResult.getProcessorResult.
• tpl: template - the template that will be run on all the files. See the example below for the shape of this template.
• tplArgs: map[string]any - a map of inputs that will be passed as-is to the template. See the example below.

Example:

//
// main.tengo, the main workflow file, somewhere in the body:
//
...
  // Here the exec command will create a file called `file.txt`. After that, a template called workdir_processor with additional arguments will run.
  run := exec.builder().
    cmd("/usr/bin/env").
    arg("bash").
    arg("-c").
    arg("echo 4 > file.txt").
    processWorkdir("result", assets.importTemplate(":workdir_processor"), {argument: "2"}).
    run()

  return {
    result: run.getProcessorResult("result")
  }
...

//
// workdir_processor.tengo, the content of the template.
//
self := import("@platforma-sdk/workflow-tengo:workdir.proc")

// readFiles helps to read a file's data.
self.readFiles(func(inputs) {
  return {
    "dataOfMyFile": "file.txt"
  }
})

// body will get several arguments:
// - inputs.workdir that is a workdir's resource.
// - inputs.dataOfMyFile with a content "4" that was got from self.readFiles
// - inputs.args that contains all arguments passed from exec.builder(). It will be {argument: "2"} in this case.
// - inputs.files with all files as resources.
self.body(func(inputs) {
  return {
    fileAsContent: inputs.dataOfMyFile + inputs.args.argument, // will be 42
    file: self.saveFile("file.txt") // will save the file with 4
  }
})

streamFile

A command to stream a file by a given file name. For example, it can be used to view live logs if the process writes them somewhere in the working directory. One can use execResult.getFileStream to get file streams.

Arguments:

• fileName: string - the name of the file to stream.

Example:

result := exec.builder().
  cmd("/usr/bin/bash").
  arg("-c").
  arg("echo 4 > output.txt; sleep 5; echo 2 > output.txt").
  streamFile("output.txt").
  run()

return {
  stream: result.getFileStream("output.txt")
}

stdoutFileName

Set stdout file name. By default it's stdout.txt. One can use getStdoutStream, getStdoutFile and getStdoutFileContent to get stdout from the result.

Arguments:

• fileName: string - the name of the stdout.

stderrFileName

Set stderr file name. By default it's stderr.txt. One can use getStderrStream, getStderrFile and getStderrFileContent to get stderr from the result.

Arguments:

• fileName: string - the name of the stderr.

saveStdoutContent

Sets stdout to be captured as a value resource. One can use getStdoutFileContent to get stdout from the result.

saveStderrContent

Sets stderr to be captured as a value resource. One can use getStderrFileContent to get stderr from the result.

printErrStreamToStdout

Redirects stderr output to stdout.

run

Executes the command.

Return:

• result: execResult - a structure that contains the execution results, see API below.

Result API

getFile

Retrieves a file.

Arguments:

• fileName: string

Return:

• file: field - a reference to the file resource.

Example:

See saveFile example.

getFileContent

Retrieves the content of a file.

Arguments:

• fileName: string

Return:

• content: field - a reference to the file content resource.

Example:

See saveFileContent example.

getFileStream

Retrieves a file stream.

Arguments:

• fileName: string

Return:

• stream: field - a reference to the file stream resource.

Example:

See streamFile example.

getFileSet

Retrieves a file set resource.

Arguments:

• fileSetName: string - name of a file set

Return:

• fileSet: field - a reference to the map resource with files

Example:

See saveFileSet example.

getFileSetContent

Retrieves a file set content resource.

Arguments:

• fileSetName: string - name of a file set

Return:

• fileSetContent: field - a reference to the map resource with files

Example:

See saveFileSetContent example.

getProcessorResult

Retrieves a processor result by the name of the processor.

Arguments:

• name: string - a name of the processor.

Return:

• result: field - a field where all the results of the processor will be.

Example:

See processWorkdir example.

getStdoutStream

Retrieves the stdout stream.

Return:

• stream: field

getStderrStream

Retrieves the stderr stream.

Return:

• stream: field

getStdoutFile

Retrieves the stdout file.

Return:

• file: field

getStderrFile

Retrieves the stderr file.

Return:

• file: field

getStdoutFileContent

Retrieves the content of the stdout file.

Return:

• content: field

getStderrFileContent

Retrieves the content of the stderr file.

Return:

• content: field