Support NetBox custom scripts

[eversC]

I'd like to be able to use custom scripts in Netbox.

I believe this will need a volume to be mounted at /opt/netbox/netbox/scripts in the deployment, and a means by which Helm users can store their scripts in a configmap (which I'm not so clear on).

[bootc]

I've been pondering this myself as well. I'm not so sure about the ConfigMap for this, it feels kind of wrong for scripts, but I don't have an immediate better answer for you either.

The old gitRepo volumes would have been more interesting for this, and there is now https://github.com/kubernetes/git-sync, but that's all a bit implementation specific. You can achieve all that today with the extraVolumeMounts / extraVolumes / extraContainers / extraInitContainers support, but it's not particularly pretty. Maybe simply adding some documentation for this would be enough?

[eversC]

Ah I wasn't aware you could do that (re: extraVolumeMounts / extraVolumes / extraContainers / extraInitContainers). That'll work for me, having it documented would be great.

[benjy44]

something good to know is that the scripts must be also on the worker pods, a working example for me:

worker:
  extraContainers:
    - name: git-custom-scripts
      image: k8s.gcr.io/git-sync/git-sync:v3.6.0
      env:
        - name: GIT_SYNC_REPO
          value: <some_repo>
        - name: GIT_SYNC_BRANCH
          value: main
        - name: GIT_SYNC_ROOT
          value: /data
        - name: GIT_SYNC_PERIOD
          value: 10m
      volumeMounts:
        - name: custom-scripts
          mountPath: /data
  extraVolumeMounts:
    - name: custom-scripts
      mountPath: /opt/netbox/netbox/scripts
  extraVolumes:
    - name: custom-scripts
      emptyDir: {}

extraContainers:
  - name: git-custom-scripts
    image: k8s.gcr.io/git-sync/git-sync:v3.6.0
    env:
      - name: GIT_SYNC_REPO
        value: <some_repo>
      - name: GIT_SYNC_BRANCH
        value: main
      - name: GIT_SYNC_ROOT
        value: /data
      - name: GIT_SYNC_PERIOD
        value: 10m
    volumeMounts:
      - name: custom-scripts
        mountPath: /data

extraVolumeMounts:
  - name: custom-scripts
    mountPath: /opt/netbox/netbox/scripts
    
extraVolumes:
  - name: custom-scripts
    emptyDir: {}

extraConfig:
  - values:
      SCRIPTS_ROOT: /opt/netbox/netbox/scripts/netbox-scripts.git/scripts

it took me a while to figure it out, as my scripts were failing with this exception when they were missing on the worker node:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.9/site-packages/rq/worker.py", line 1061, in perform_job
    rv = job.perform()
  File "/opt/netbox/venv/lib/python3.9/site-packages/rq/job.py", line 821, in perform
    self._result = self._execute()
  File "/opt/netbox/venv/lib/python3.9/site-packages/rq/job.py", line 844, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/opt/netbox/netbox/extras/scripts.py", line 438, in run_script
    script = get_script(module, script_name)()
TypeError: 'NoneType' object is not callable

[LeoColomb]

From #148 by @florianschendel

Support storage config for scripts as for reports

How it should look in the values file

## Storage configuration for Scripts
scriptsPersistence:
  enabled: true
  ##
  ## Existing claim to use
  existingClaim: ""
  ## Existing claim's subPath to use, e.g. "media" (optional)
  subPath: ""
  ##
.....

My plan is to use scripts in combination with your helm chart. I know i could also create an extra container but it make no sense for me because the Reports/Scripts are standard features of Netbox.

There is already an open issue #10. My opinion is that the extra container solution is for special use cases.

[moonrail]

Hmm, this has become significantly more complicated with NetBox 4 as (at least on my setups) Scripts and Reports are not being autodiscovered as with 3.7.8 and before.

With 3.7.8 we could - like described in this issue - mount/copy our Scripts to /opt/netbox/netbox/scripts and Reports to /opt/netbox/netbox/reports and they were usable.

With 4.1.3 the Reports feature has been removed and a single directory /opt/netbox/netbox/scripts is being used for both.
Here is the catch:

• Scripts provided in this directory are ignored
• this directory is required by NetBox to be writable, otherwise a uncatched OSError is raised on creation/deletion of extras.models.Script
• NetBox will write files into this directory when a User creates a "Script" via UI/API
  • the content of a file is sourced from the database (core.models.ManagedFile and its inheriting models)
  • the files in the database are created from a DataSource, that a User has to create
    • this DataSource can source from Git, S3 oder local
• only web nodes of NetBox will write these files, but workers will use them
  • therefore all web nodes need a ReadWriteMany storage
  • therefore all worker nodes have to use the same storage as the web nodes
• DataSources cannot be synced automatically and will not be sinced to e.g. local filesystem changes without User interaction
• Scripts based on DataSources will differ from file content in the database, unless "Auto Sync" is enabled on them (why does this "feature" exist?)

I cannot see a clean solution without ReadWriteMany storage for all nodes. Every other solution I can think of is somewhat of a hack or may break with the next change of NetBox.

Kind of weird, that this feature was implemented this way. I think I get the intend, but it looks very much designed for single-node-instances of NetBox with local running rqworkers. Without any sort of central writable network storage this feature is broken for cluster installations.

If all files are in the database already, why do they have to be written to the filesystem beforehand by the web process? Why do not all workers upon job execution just create a temporary file based on the contents in the database? Or invoke the scripts in memory only.

Also this leads to local DataSources effecively being tripled. The file resides in the source path, in the database row and in /opt/netbox/netbox/scripts.

[riconem]

Is there anybody working on this?