Skip to content

Latest commit

 

History

History
64 lines (55 loc) · 3.51 KB

TODO.org

File metadata and controls

64 lines (55 loc) · 3.51 KB

previous

enable fink-test to run test on fink releases (currently it runs only on branch master tips)

hdfs operator management

limit and request for memory: monitor issue stackabletech/hdfs-operator#625

open issue: zkfc on datanode is not compliant with memory setting

In the example below memory limit is 256Mi for nameNode in hdfscluster CR, but it become 768Mi in each related pod because the `zkfs` container is not impacted by the CR configuration. This should be fixed because it prevents running the setup on CI platforms with low memory like Github Action for instances.

kubectl get -n hdfs hdfscluster simple-hdfs -o yaml -o jsonpath -o=jsonpath=’{.spec.nameNodes.config.resources}’ {“cpu”:{“min”:”0”},”memory”:{“limit”:”256Mi”}}

kubectl describe nodes | grep namenode hdfs simple-hdfs-namenode-default-0 100m (0%) 1400m (1%) 768Mi (0%) 768Mi (0%) 34m hdfs simple-hdfs-namenode-default-1 100m (0%) 1400m (1%) 768Mi (0%) 768Mi (0%) 31m

kubectl get pods -n hdfs simple-hdfs-namenode-default-0 -o jsonpath -o=jsonpath=’{.spec.containers[1].name}’ zkfc

kubectl get pods -n hdfs simple-hdfs-namenode-default-0 -o jsonpath -o=jsonpath=’{.spec.containers[1].resources}’ | jq { “limits”: { “cpu”: “400m”, “memory”: “512Mi” }, “requests”: { “cpu”: “100m”, “memory”: “512Mi” } }

management of argoCD default values (jqpath expression): monitor issue stackabletech/hdfs-operator#626

open issue: be able to run only one dataNode on CI

Add helm option on HDFS cpu.min (also for operators!)

Move fink image to docker.stackable.tech/stackable/hadoop:3.3.6-stackable24.11.

DELAYED BECAUSE IT NOT BLOCKS BUT WARN create topic in distribute before sending alerts in order to avoid error below: https://fink-broker.slack.com/archives/D03KJ390F17/p1692008729660549

Du coup ça fonctionne avec un compte utilisateur, par contre j’ai pas activé les autorisations dans kafka car le fink-alert-simulator aurait pu plus écrire dans le topic sans authentification. 12 h 28 J’ai maintenant ce message d’erreur: 23/08/14 10:26:52 WARN NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 29 : {fink_simbad_grav_candidates_ztf=LEADER_NOT_AVAILABLE} 12 h 32 En fait c’est du au fait que le topic existe pas, ça fonctionne si on relance lae job distribute… 12 h 33 Tu crois qu’on pourrais pré-créer les topic pour éviter ce problème @JulienPeloton ?

Enable authZ in kafka (require authN setup in fink-alert-simulator)

move nodeport to internal for svc kafka-cluster-kafka-external-bootstrap

run code-check.sh in CI

manage dependencies

What to do with:

  1. hbase-spark-hbase2.4_spark3_scala2.12_hadoop3.2.jar

hbase-spark-protocol-shaded-hbase2.4_spark3_scala2.12_hadoop3.2.jar which are both in k8s-spark-py/custom and fink-broker/libs (cf. FINK_JARS) cf. Julien are they required?

  1. custom/jars/commons-pool2-2.6.2.jar which was in k8s-spark-py/custom

test removal of options below whith useing hdfs

  • –conf spark.driver.extraJavaOptions=”-Divy.cache.dir=/tmp -Divy.home=/tmp” \ –conf spark.hadoop.fs.s3a.path.style.access=true \
  • –conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \