Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ingestion Failure: Error Related to datahub-kafka Module #526

Open
Haebuk opened this issue Jan 6, 2025 · 0 comments
Open

Ingestion Failure: Error Related to datahub-kafka Module #526

Haebuk opened this issue Jan 6, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@Haebuk
Copy link

Haebuk commented Jan 6, 2025 •
edited
Loading

Description
The ingestion process, which was previously working fine, has been failing with the following error since December 28, 2024. It was functioning correctly until December 27, and the same error has been recurring since then.

Error Log

~~~~ Execution Summary - RUN_INGEST ~~~~
Execution finished with errors.
{'exec_id': '11f6a05f-fedc-4bdb-8378-187e5fb3ec48',
 'infos': ['2024-12-27 15:00:04.685523 INFO: Starting execution for task with name=RUN_INGEST',
           "2024-12-27 15:00:51.613868 INFO: Failed to execute 'datahub ingest', exit code 1",
           '2024-12-27 15:00:51.614052 INFO: Caught exception EXECUTING task_id=11f6a05f-fedc-4bdb-8378-187e5fb3ec48, name=RUN_INGEST, '
           'stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 140, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 282, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"],
 'errors': []}

~~~~ Ingestion Logs ~~~~
...
[2024-12-27 15:00:50,078] ERROR    {datahub.entrypoints:201} - Command failed: Failed to find a registered sink for type datahub-kafka: datahub-kafka is disabled; try running: pip install 'acryl-datahub[datahub-kafka]'
...
ModuleNotFoundError: No module named 'httpx'
...
datahub.configuration.common.ConfigurationError: datahub-kafka is disabled; try running: pip install 'acryl-datahub[datahub-kafka]'
...

helm history

29              Mon Jul 22 17:00:07 2024        superseded      datahub-0.4.19  0.13.2          Upgrade complete                                                     
30              Fri Jan  3 17:41:02 2025        failed          datahub-0.4.19  0.13.2          Upgrade "datahub" failed: context canceled                           
31              Fri Jan  3 17:44:43 2025        superseded      datahub-0.4.19  0.13.2          Upgrade complete                                                     
32              Mon Jan  6 13:46:45 2025        superseded      datahub-0.4.19  0.13.2          Upgrade complete                                                     
33              Mon Jan  6 13:58:16 2025        superseded      datahub-0.4.19  0.13.2          Upgrade complete                                                     
34              Mon Jan  6 14:30:51 2025        failed          datahub-0.4.39  0.14.1          Upgrade "datahub" failed: context canceled                           
35              Mon Jan  6 14:31:03 2025        failed          datahub-0.4.39  0.14.1          Upgrade "datahub" failed: pre-upgrade hooks failed: 1 error occurr...
36              Mon Jan  6 14:42:19 2025        failed          datahub-0.4.39  0.14.1          Upgrade "datahub" failed: context canceled                           
37              Mon Jan  6 14:46:07 2025        failed          datahub-0.4.39  0.14.1          Upgrade "datahub" failed: context canceled                           
38              Mon Jan  6 14:48:31 2025        deployed        datahub-0.4.39  0.14.1          Upgrade complete

Additional Information
There have been no changes between the deployment on December 22, 2024, and the deployment on January 3, 2025.
The datahub-kafka module is disabled, and the httpx module appears to be missing.

Here is my override-values.yaml

datahub-gms:
  enabled: true
  image:
    repository: acryldata/datahub-gms
  resources:
    limits:
      memory: 8Gi
    requests:
      cpu: 100m
      memory: 8Gi
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]
  extraEnvs:
    - name: SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_PARTITION_FETCH_BYTES
      value: "20971520"
    - name: SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
      value: "20971520"
    - name: ES_BULK_REQUESTS_LIMIT
      value: "100"
  nodeSelector:
    env: "data-prod"
    arch: "arm64"
    category: "normal"
    node: "karpenter"
  tolerations:
    - key: "service"
      operator: "Equal"
      value: "arm64"
      effect: "NoSchedule"

datahub-frontend:
  enabled: true
  image:
    repository: acryldata/datahub-frontend-react
  resources:
    limits:
      memory: 1400Mi
    requests:
      cpu: 100m
      memory: 1400Mi
  ingress:
    enabled: true
    hosts:
      - host: "[MASKED]"
        paths: ["/"]
    className: "nginx"
  defaultUserCredentials:
    randomAdminPassword: true
  service:
    type: ClusterIP
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]

acryl-datahub-actions:
  enabled: true
  image:
    repository: acryldata/datahub-actions
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 1
      memory: 4Gi
  extraVolumeMounts:
    - name: executor-config
      mountPath: /etc/datahub/actions/system/conf
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: executor-config
      configMap:
        name: datahub-actions-config
        defaultMode: 0644
    - name: msk-certs
      secret:
        secretName: [MASKED]
  extraEnvs:
    - name: SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL
      value: "SSL"
    - name: SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION
      value: "/mnt/datahub/certs/kafka.client.truststore.jks"
    - name: DATAHUB_ACTIONS_SLACK_CHANNEL
      value: [MASKED]
    - name: DATAHUB_ACTIONS_SLACK_BOT_TOKEN
      valueFrom:
        secretKeyRef:
          name: datahub-slack-secrets
          key: [MASKED]
    - name: DATAHUB_ACTIONS_SLACK_SIGNING_SECRET
      valueFrom:
        secretKeyRef:
          name: datahub-slack-secrets
          key: [MASKED]

datahub-mae-consumer:
  image:
    repository: acryldata/datahub-mae-consumer
  resources:
    limits:
      memory: 1536Mi
    requests:
      cpu: 100m
      memory: 1536Mi
  extraEnvs:
    - name: SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_PARTITION_FETCH_BYTES
      value: "20971520"
    - name: SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
      value: "20971520"
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]

datahub-mce-consumer:
  image:
    repository: acryldata/datahub-mce-consumer
  resources:
    limits:
      memory: 1536Mi
    requests:
      cpu: 100m
      memory: 1536Mi
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]
  extraEnvs:
    - name: SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_PARTITION_FETCH_BYTES
      value: "20971520"
    - name: SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
      value: "20971520"

datahub-ingestion-cron:
  enabled: false
  image:
    repository: acryldata/datahub-ingestion
    # tag: "v0.11.0" # defaults to .global.datahub.version

elasticsearchSetupJob:
  enabled: true
  image:
    repository: acryldata/datahub-elasticsearch-setup
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 500m
      memory: 512Mi

kafkaSetupJob:
  enabled: true
  image:
    repository: acryldata/datahub-kafka-setup
  resources:
    limits:
      cpu: 1
      memory: 4Gi
    requests:
      cpu: 1
      memory: 4Gi
  extraEnvs:
    - name: SPRING_KAFKA_SECURITY_PROTOCOL
      value: "SSL"
    - name: SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_PARTITION_FETCH_BYTES
      value: "20971520"
    - name: SPRING_KAFKA_PRODUCER_PROPERTIES_MAX_REQUEST_SIZE
      value: "20971520"
    - name: KAFKA_HEAP_OPTS
      value: "-Xmx2g -Xms1g"
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]

datahubUpgrade:
  cleanupJob:
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 512Mi
  restoreIndices:
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 512Mi
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]

datahubSystemUpdate:
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 500m
      memory: 512Mi
  extraVolumeMounts:
    - name: msk-certs
      mountPath: /mnt/datahub/certs
      readOnly: true
  extraVolumes:
    - name: msk-certs
      secret:
        secretName: [MASKED]

global:
  elasticsearch:
    host: "[MASKED]"
    port: "9243"
    skipcheck: "false"
    insecure: "false"
    useSSL: "true"
    indexPrefix: "datahub"
    auth:
      username: [MASKED]
      password:
        secretRef: elasticsearch-secrets
        secretKey: [MASKED]

  kafka:
    bootstrap:
      server: "[MASKED]"
    zookeeper:
      server: "[MASKED]"
    partitions: 3
    replicationFactor: 3
    schemaregistry:
      type: KAFKA
      url: "[MASKED]"
  springKafkaConfigurationOverrides:
    security.protocol: "SSL"
    ssl.truststore.location: "/mnt/datahub/certs/kafka.client.truststore.jks"
    ssl.truststore.type: "JKS"
    ssl.truststore.password: [MASKED]
    ssl.endpoint.identification.algorithm: ""

  sql:
    datasource:
      host: "[MASKED]"
      hostForMysqlClient: "[MASKED]"
      port: "3306"
      url: "[MASKED]"
      driver: "com.mysql.cj.jdbc.Driver"
      username: [MASKED]
      password:
        secretRef: mysql-secrets
        secretKey: [MASKED]

Request
Investigate the cause of the datahub-kafka module being disabled and suggest a resolution.
Resolve the missing httpx module issue.

Please look into this issue and provide a solution. Thank you.
@Haebuk Haebuk added the bug Something isn't working label Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Haebuk