Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Cassandra Db fails with foreign keys' error #97

Open
vga91 opened this issue Aug 19, 2024 · 0 comments
Open

The Cassandra Db fails with foreign keys' error #97

vga91 opened this issue Aug 19, 2024 · 0 comments

Comments

@vga91
Copy link

vga91 commented Aug 19, 2024

Expected Behavior (Mandatory)

The export command should work

Actual Behavior (Mandatory)

Foreign key error

How to Reproduce the Problem

docker-compose.yml

version: '3.8'

services:
  etl:
    hostname: etl
    image: vga91/etl:1.0.0
    networks:
      - my_network
    volumes:
      - ./my_data:/app/lib

  cassandra:
    hostname: cassandra
    image: cassandra:4
    environment:
      - CASSANDRA_CLUSTER_NAME=TestCluster
      - CASSANDRA_DC=dc1
      - CASSANDRA_RACK=rack1
      - CASSANDRA_ENDPOINT_SNITCH=SimpleSnitch
      - CASSANDRA_SEEDS=cassandra
      - CASSANDRA_START_RPC=true
    networks:
      - my_network
    ports:
      - "9042:9042"
    volumes:
      - /var/lib/cassandra
  cassandrasample:
    hostname: cassandrasample
    image: cassandra:4
    networks:
      - my_network
    depends_on:
      - cassandra
    restart: "no"
    entrypoint: ["/sample_data.sh"]
    volumes:
      - ./sample_data.sh:/sample_data.sh
    
networks:
  my_network:
    driver: bridge

sample_data.sh

#!/usr/bin/env bash

until printf "" 2>>/dev/null >>/dev/tcp/cassandra/9042; do 
    sleep 5;
    echo "Waiting for cassandra...";
done

echo "Creating keyspace and tables..."
cqlsh cassandra -e "CREATE KEYSPACE IF NOT EXISTS test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};"
cqlsh cassandra -e "CREATE TABLE IF NOT EXISTS test.test (sensor_id uuid, registered_at timestamp, temperature int, location text, PRIMARY KEY ((sensor_id), registered_at)) WITH CLUSTERING ORDER BY (registered_at DESC);"
cqlsh cassandra -e "CREATE TABLE IF NOT EXISTS test.sensors_locations (bucket text, location text, sensor_id uuid, PRIMARY KEY((bucket), location));"

cqlsh cassandra -e "insert into test.sensors_locations (bucket, location, sensor_id) values ('default', 'kitchen', 99051fe9-6a9c-46c2-b949-38ef78858dd0);";
cqlsh cassandra -e "insert into test.sensors_locations (bucket, location, sensor_id) values ('default', 'bedroom', 99051fe9-6a9c-46c2-b949-38ef78858dd1);";

#while true; do
    echo "Writing sample data...";
    cqlsh cassandra -e "insert into test.test (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, toTimestamp(now()), $(shuf -i 18-32 -n 1), 'kitchen');";
    cqlsh cassandra -e "insert into test.test (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd1, toTimestamp(now()), $(shuf -i 12-40 -n 1), 'bedroom');";
    echo "Written sample data";
#    sleep 5;
#done

Execute the following commands:

bin/neo4j-etl",
                    "generate-metadata-mapping",
                    "--rdbms:url",
                        jdbc:cassandra://cassandra:9042/test?localdatacenter=dc1,
                    "--rdbms:user",
                        username,
                    "--rdbms:password",
                        password,
                    "--output-mapping-file",
                    "/tmp/mapping.json",
                    "--csv-directory",
                    "/tmp/northwind"

and then

"bin/neo4j-etl",
                        "export",
                        "--rdbms:url",
                        jdbc:cassandra://cassandra:9042/test?localdatacenter=dc1,
                        "--rdbms:user",
                        username,
                        "--rdbms:password",
                        password,
                        "--using",
                        "cypher:batch",
                        "--csv-directory",
                        "/tmp/northwind",
                        "--mapping-file",
                        "/tmp/mapping.json",
                        "--neo4j:url",
                        "bolt://neo4j:7687",
                        "--neo4j:user",
                        "neo4j",
                        "--neo4j:password",
                        "password",
                        "--unwindBatchSize",
                        "1000",
                        "--txBatchSize",
                        "1000",
                        "--quote",
                        "\"",
                        "--force"

Error:

- Crawling tables
- Retrieving tables
- Retrieving tables for schema <test>
- Processed 2 rows for <retrieveTables>
- Retrieved 2 tables
- Retrieving table columns 
- Retrieving foreign keys
- Command failed due to error (SQLException: Could not retrieve foreign keys for table test.sensors_locations: The Cassandra implementation does not support this method.). Rerun with --debug flag for detailed diagnostic information.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant