bug: hidden files during data injection

[andrewrisse]

Environment

Device and OS: Apple M3 Max, macOS Sequoia 15.1.1
App version: Zarf 0.32.4, UDS 0.19.0
Kubernetes distro being used:
k3d version v5.7.4
k3s version v1.30.4-k3s1 (default)

Steps to reproduce

While using data injection to add files to my pod when utilizing arm64 architecture, hidden files with the convention: ._* are being added to the target directory. During deployment, this results in this error message and a failed UDS deployment:

._.                                                                                                                                                      
   ./                                                                                                                                                       
   ./._daily.cvd                                                                                                                                                 
   ./daily.cvd                                                                                                                                                  
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   ./._daily-27509.cdiff                                                                                                                                             
   ./daily-27509.cdiff                                                                                                                                              
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   ./._bytecode-335.cdiff                                                                                                                                             
   ./bytecode-335.cdiff                                                                                                                                              
   ./._main.cvd                                                                                                                                                  
   ./main.cvd                                                                                                                                                   
   ./._dns.txt                                                                                                                                                  
   ./dns.txt                                                                                                                                                   
   ./._bytecode.cvd                                                                                                                                                
   ./bytecode.cvd                                                                                                                                                 
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance'                                                                                                             
   ./._main-62.cdiff                                                                                                                                               
   ./main-62.cdiff                                                                                                                                                
   tar: .: Cannot utime: Operation not permitted                                                                                                                                 
   tar: .: Cannot change mode to rwxr-xr-x: Operation not permitted                                                                                                                        
   tar: Exiting with failure status due to previous errors                                                                                                                            
   command terminated with exit code 2                                                                                                                                      
     pod/clamav-66df89d4fb-gfpgr condition met

I am attempting to inject these files into a pod runnning ClamAV and these hidden files cause ClamAV to break because it only expects to find the actual files (e.g. main.cvd, not ._main.cvd).

This bug fix request is to reconfigure Zarf to not preserve xattrs for data injection to prevent the aforementioned deployment error and unexpected behavior (addition of hidden files).

1. Example zarf.yaml that triggers this:

components:
  - name: clamav
    required: true
    description: "Deploy ClamAV"
    import:
      path: common
    only:
      flavor: upstream
    charts:
      - name: clamav
        valuesFiles:
          - values/upstream-values.yaml
    images:
      - docker.io/clamav/clamav-debian:1.4.1
    dataInjections:
      - source: clamav-definitions
        target:
          namespace: clamav
          selector: app=clamav-12.5.0 
          container: clamav
          path: /var/lib/clamav

Expected result

FIles copied into target directory match the source exactly, no hidden files added
No error messages during UDS deployment

Actual Result

Extra files, error messages, failed deployment, broken app

Severity/Priority

Medium - blocking arm64 deployment for uds-package-clamav

Additional Context

From @bburky :

It seems like zarf uses tar under the hood to create a tarball for data injection?
Those files are one way to represent xattrs in a tarball. macOS uses xattrs for various things, but notably com.apple.quarantine is used to mark files downloaded from the web, so the OS can ask "are you sure you want to do whatever with this possibly untrusted file?" (this is the equivalent of Windows' Mark of the Web)
There's probably no reason to preserve xattrs for data injection, seems like a reasonable zarf feature request/bug fix to not do this.
Oddly, if I test on the command line tar czf foo.tar.gz ~/Downloads/something-youv've-downloaded I seem to get native tar xattrs and not ._ files, but if I use tar --no-xattrs -czvf ... I indeed get the files you mention. (you need to check the created tar file on Linux, tar tzf on macOS won't even list the ._ files it seems)
It seems like zarf currently runs tar -c -z -f - on the host (your mac) to create the tarball:
https://docs.zarf.dev/ref/examples/kiwix/

zarf/src/pkg/cluster/data.go

Lines 34 to 161 in d587539

	// HandleDataInjection waits for the target pod(s) to come up and inject the data into them
	// todo: this currently requires kubectl but we should have enough k8s work to make this native now.
	func (c *Cluster) HandleDataInjection(ctx context.Context, data v1alpha1.ZarfDataInjection, componentPath *layout.ComponentPaths, dataIdx int) error {
	l := logger.From(ctx)
	injectionCompletionMarker := filepath.Join(componentPath.DataInjections, config.GetDataInjectionMarker())
	if err := os.WriteFile(injectionCompletionMarker, []byte("🦄"), helpers.ReadWriteUser); err != nil {
	return fmt.Errorf("unable to create the data injection completion marker: %w", err)
	}
	tarCompressFlag := ""
	if data.Compress {
	tarCompressFlag = "-z"
	}
	// Pod filter to ensure we only use the current deployment's pods
	podFilterByInitContainer := func(pod corev1.Pod) bool {
	b, err := json.Marshal(pod)
	if err != nil {
	return false
	}
	// Look everywhere in the pod for a matching data injection marker
	return strings.Contains(string(b), config.GetDataInjectionMarker())
	}
	// Get the OS shell to execute commands in
	shell, shellArgs := exec.GetOSShell(v1alpha1.Shell{Windows: "cmd"})
	if _, _, err := exec.Cmd(shell, append(shellArgs, "tar --version")...); err != nil {
	return fmt.Errorf("unable to execute tar, ensure it is installed in the $PATH: %w", err)
	}
	message.Debugf("Attempting to inject data into %s", data.Target)
	l.Debug("performing data injection", "target", data.Target)
	source := filepath.Join(componentPath.DataInjections, filepath.Base(data.Target.Path))
	if helpers.InvalidPath(source) {
	// The path is likely invalid because of how we compose OCI components, add an index suffix to the filename
	source = filepath.Join(componentPath.DataInjections, strconv.Itoa(dataIdx), filepath.Base(data.Target.Path))
	if helpers.InvalidPath(source) {
	return fmt.Errorf("could not find the data injection source path %s", source)
	}
	}
	// Wait until the pod we are injecting data into becomes available
	target := podLookup{
	Namespace: data.Target.Namespace,
	Selector: data.Target.Selector,
	Container: data.Target.Container,
	}
	waitCtx, waitCancel := context.WithTimeout(ctx, 90*time.Second)
	defer waitCancel()
	pods, err := waitForPodsAndContainers(waitCtx, c.Clientset, target, podFilterByInitContainer)
	if err != nil {
	return err
	}
	// Inject into all the pods
	for _, pod := range pods {
	// Try to use the embedded kubectl if we can
	zarfCommand, err := utils.GetFinalExecutableCommand()
	kubectlBinPath := "kubectl"
	if err != nil {
	message.Warnf("Unable to get the zarf executable path, falling back to host kubectl: %s", err)
	l.Warn("unable to get the zarf executable path, falling back to host kubectl", "error", err)
	} else {
	kubectlBinPath = fmt.Sprintf("%s tools kubectl", zarfCommand)
	}
	kubectlCmd := fmt.Sprintf("%s exec -i -n %s %s -c %s ", kubectlBinPath, data.Target.Namespace, pod.Name, data.Target.Container)
	// Note that each command flag is separated to provide the widest cross-platform tar support
	tarCmd := fmt.Sprintf("tar -c %s -f -", tarCompressFlag)
	untarCmd := fmt.Sprintf("tar -x %s -v -f - -C %s", tarCompressFlag, data.Target.Path)
	// Must create the target directory before trying to change to it for untar
	mkdirCmd := fmt.Sprintf("%s -- mkdir -p %s", kubectlCmd, data.Target.Path)
	if err := exec.CmdWithPrint(shell, append(shellArgs, mkdirCmd)...); err != nil {
	return fmt.Errorf("unable to create the data injection target directory %s in pod %s: %w", data.Target.Path, pod.Name, err)
	}
	cpPodCmd := fmt.Sprintf("%s -C %s . | %s -- %s",
	tarCmd,
	source,
	kubectlCmd,
	untarCmd,
	)
	// Do the actual data injection
	if err := exec.CmdWithPrint(shell, append(shellArgs, cpPodCmd)...); err != nil {
	return fmt.Errorf("could not copy data into the pod %s: %w", pod.Name, err)
	}
	// Leave a marker in the target container for pods to track the sync action
	cpPodCmd = fmt.Sprintf("%s -C %s %s | %s -- %s",
	tarCmd,
	componentPath.DataInjections,
	config.GetDataInjectionMarker(),
	kubectlCmd,
	untarCmd,
	)
	if err := exec.CmdWithPrint(shell, append(shellArgs, cpPodCmd)...); err != nil {
	return fmt.Errorf("could not save the Zarf sync completion file after injection into pod %s: %w", pod.Name, err)
	}
	}
	// Do not look for a specific container after injection in case they are running an init container
	podOnlyTarget := podLookup{
	Namespace: data.Target.Namespace,
	Selector: data.Target.Selector,
	}
	// Block one final time to make sure at least one pod has come up and injected the data
	// Using only the pod as the final selector because we don't know what the container name will be
	// Still using the init container filter to make sure we have the right running pod
	_, err = waitForPodsAndContainers(ctx, c.Clientset, podOnlyTarget, podFilterByInitContainer)
	if err != nil {
	return err
	}
	// Cleanup now to reduce disk pressure
	err = os.RemoveAll(source)
	if err != nil {
	return err
	}
	// Return to stop the loop
	return nil
	}

I think this uses whichever tar is on the $PATH (which maybe macOS's tar, but could be something out of homebrew too I think). The exact options seem to vary by which tar, macos wants --no-mac-metadata I think, but for other tar variants --no-attrs may be enough?
The safest fix may be to use golang's built in archive/tar and make sure it does the right thing.