Merge branch 'xlab-uiuc:main' into porting

.github/ISSUE_TEMPLATE/alarm-inspection-report.yaml

@@ -0,0 +1,29 @@
+name: Alarm Inspection Report
+description: An analysis report for the alarms produced by Acto
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: What happened?
+      description: |
+        Why did Acto raise this alarm? What happened in the state transition? Why Acto’s oracles raised an alarm?
+    validations:
+      required: true
+
+  - type: textarea
+    id: root-cause
+    attributes:
+      label: What did you expect to happen?
+      description: |
+        Why did the operator behave in this way? Please find the exact block in the operator source code resulting in the behavior.
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: Root Cause
+      description: |
+        If it is a true alarm, how to fix it in the operator code? If it is a false alarm, how to fix it in Acto code?
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

Makefile

@@ -4,4 +4,4 @@ lib:
 
 clean:
 	(cd acto/k8s_util/lib && make clean)
-	(cd ssa && make)
+	(cd ssa && make clean)

README.md

@@ -1,4 +1,4 @@
-# Acto: Push-Button End-to-End Testing of Kubernetes Operators/Controllers
+# Acto: Push-Button End-to-End Testing of Kubernetes Operators and Controllers
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![Regression Test](https://github.com/xlab-uiuc/acto/actions/workflows/unittest.yaml/badge.svg)](https://github.com/xlab-uiuc/acto/actions/workflows/unittest.yaml)
 [![End-to-End Test](https://github.com/xlab-uiuc/acto/actions/workflows/e2e-test.yml/badge.svg)](https://github.com/xlab-uiuc/acto/actions/workflows/e2e-test.yml)

acto/__main__.py

@@ -51,6 +51,12 @@
     default=1,
     help="Number of concurrent workers to run Acto with",
 )
+parser.add_argument(
+    "--helper_crd",
+    dest="helper_crd",
+    type=str,
+    help="helper crd file to test on",
+)
 parser.add_argument(
     "--num-cases",
     dest="num_cases",
@@ -133,7 +139,7 @@
     cluster_runtime="KIND",
     preload_images_=None,
     context_file=context_cache,
-    helper_crd=None,
+    helper_crd=args.helper_crd,
     num_workers=args.num_workers,
     num_cases=args.num_cases,
     dryrun=args.dryrun,

acto/deploy.py

@@ -81,8 +81,7 @@ def deploy(self,
         # Run the steps in the deploy config one by one
         for step in self._deploy_config.steps:
             if step.apply:
-                args = ["apply", "--server-side", "-f", step.apply.file,
-                        "--context", context_name]
+                args = ["apply", "--server-side", "-f", step.apply.file]
 
                 # Use the namespace from the argument if the namespace is delegated
                 # If the namespace from the config is explicitly specified,

acto/engine.py

@@ -933,26 +933,26 @@ def __learn(self, context_file, helper_crd, analysis_only=False):
                 os.path.expanduser("~"), ".kube", learn_context_name
             )
 
-            while True:
-                self.cluster.restart_cluster("learn", learn_kubeconfig)
-                namespace = (
-                    get_yaml_existing_namespace(self.deploy.operator_yaml)
-                    or CONST.ACTO_NAMESPACE
-                )
-                self.context["namespace"] = namespace
-                kubectl_client = KubectlClient(
-                    learn_kubeconfig, learn_context_name
-                )
-                deployed = self.deploy.deploy_with_retry(
-                    learn_kubeconfig,
-                    learn_context_name,
-                    kubectl_client=kubectl_client,
-                    namespace=namespace,
+            self.cluster.restart_cluster("learn", learn_kubeconfig)
+            namespace = (
+                get_yaml_existing_namespace(self.deploy.operator_yaml)
+                or CONST.ACTO_NAMESPACE
+            )
+            self.context["namespace"] = namespace
+            kubectl_client = KubectlClient(learn_kubeconfig, learn_context_name)
+            deployed = self.deploy.deploy_with_retry(
+                learn_kubeconfig,
+                learn_context_name,
+                kubectl_client=kubectl_client,
+                namespace=namespace,
+            )
+            if not deployed:
+                raise RuntimeError(
+                    f"Failed to deploy operator due to max retry exceed"
                 )
-                if deployed:
-                    break
-            apiclient = kubernetes_client(learn_kubeconfig, learn_context_name)
 
+            apiclient = kubernetes_client(learn_kubeconfig, learn_context_name)
+            logger.debug("helper crd path is %s", helper_crd)
             self.context["crd"] = process_crd(
                 apiclient,
                 KubectlClient(learn_kubeconfig, learn_context_name),

acto/input/input.py

@@ -251,6 +251,11 @@ def __init__(
             )
         )
 
+        for base_schema, k8s_schema_name in self.full_matched_schemas:
+            base_schema.attributes |= (
+                property_attribute.PropertyAttribute.Mapped
+            )
+
         # Apply custom property attributes based on the property_attribute module
         self.apply_custom_field()
 
@@ -288,7 +293,9 @@ def generate_test_plan(
 
         normal_testcases = {}
 
-        test_cases = get_testcases(self.root_schema, self.full_matched_schemas)
+        test_cases = get_testcases(
+            self.get_schema_by_path(self.mount), self.full_matched_schemas
+        )
 
         num_test_cases = 0
         num_run_test_cases = 0

acto/input/test_generators/stateful_set.py

@@ -62,7 +62,7 @@ def replicas_tests(schema: IntegerSchema) -> list[TestCase]:
 
 
 @test_generator(
-    property_name="statefulSetUpdateStrategy", priority=Priority.SEMANTIC
+    property_name="updateStrategy", priority=Priority.SEMANTIC
 )
 def stateful_set_update_strategy_tests(schema: ObjectSchema) -> list[TestCase]:
     """Generate test cases for StatefulSetUpdateStrategy field"""

acto/k8s_util/lib/Makefile

@@ -3,5 +3,5 @@ k8sutil:
 	gcc test.c -o test ./k8sutil.so
 
 clean:
-	rm ./k8sutil.so
-	rm ./test
+	rm -f ./k8sutil.so
+	rm -f ./test

acto/kubernetes_engine/kind.py

@@ -110,7 +110,7 @@ def create_cluster(self, name: str, kubeconfig: str):
 
         cmd.extend(["--image", f"kindest/node:{self._k8s_version}"])
 
-        p = subprocess.run(cmd, check=True)
+        p = subprocess.run(cmd, check=False)
         i = 0
         while p.returncode != 0:
             if i == 3:
@@ -122,7 +122,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         try:
             kubernetes.config.load_kube_config(
@@ -154,7 +154,7 @@ def load_images(self, images_archive_path: str, name: str):
         else:
             logging.error("Missing cluster name for kind load")
 
-        p = subprocess.run(cmd + [images_archive_path], check=True)
+        p = subprocess.run(cmd + [images_archive_path], check=False)
         if p.returncode != 0:
             logging.error("Failed to preload images archive")
 
@@ -171,7 +171,7 @@ def delete_cluster(self, name: str, kubeconfig: str):
         else:
             raise RuntimeError("Missing kubeconfig for kind create")
 
-        while subprocess.run(cmd, check=True).returncode != 0:
+        while subprocess.run(cmd, check=False).returncode != 0:
             continue
 
     def get_node_list(self, name: str):

acto/kubernetes_engine/minikube.py

@@ -69,7 +69,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
 
         i = 0
@@ -83,7 +83,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         # csi driver
         cmd = ["minikube", "addons", "disable", "storage-provisioner"]
@@ -93,7 +93,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
         i = 0
         print(cmd)
@@ -107,7 +107,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         cmd = ["minikube", "addons", "disable", "default-storageclass"]
         cmd.extend(["--profile", name])
@@ -116,7 +116,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
         i = 0
         print(cmd)
@@ -130,15 +130,15 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         cmd = ["minikube", "addons", "enable", "volumesnapshots"]
         cmd.extend(["--profile", name])
         p = subprocess.run(
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
         i = 0
         print(cmd)
@@ -152,15 +152,15 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         cmd = ["minikube", "addons", "enable", "csi-hostpath-driver"]
         cmd.extend(["--profile", name])
         p = subprocess.run(
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
         i = 0
         print(cmd)
@@ -174,7 +174,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         cmd = [
             "kubectl",
@@ -188,7 +188,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             cmd,
             stdout=subprocess.DEVNULL,
             stderr=subprocess.DEVNULL,
-            check=True,
+            check=False,
         )
         i = 0
         print(cmd)
@@ -202,7 +202,7 @@ def create_cluster(self, name: str, kubeconfig: str):
             i += 1
             self.delete_cluster(name, kubeconfig)
             time.sleep(5)
-            p = subprocess.run(cmd, check=True)
+            p = subprocess.run(cmd, check=False)
 
         # minikube mount
         cmd = ["minikube", "mount", "profile/data:/tmp/profile"]
@@ -239,7 +239,7 @@ def load_images(self, images_archive_path: str, name: str):
         else:
             logging.error("Missing cluster name for minikube load")
 
-        p = subprocess.run(cmd + [images_archive_path], check=True)
+        p = subprocess.run(cmd + [images_archive_path], check=False)
         if p.returncode != 0:
             logging.error("Failed to preload images archive")
 
@@ -257,7 +257,7 @@ def delete_cluster(self, name: str, kubeconfig: str):
         else:
             raise RuntimeError("Missing kubeconfig for minikube create")
 
-        while subprocess.run(cmd, check=True).returncode != 0:
+        while subprocess.run(cmd, check=False).returncode != 0:
             continue
 
         os.environ.pop("KUBECONFIG", None)

acto/parse_log/parse_log.py

@@ -45,6 +45,14 @@
 logrus_regex += r'\s*$'
 # this is semi-auto generated by copilot, holy moly
 
+# This one is similar to logr_special_regex and logrus_regex, but with some differences
+# 2024-03-05T10:07:17Z	ERROR	GrafanaReconciler	reconciler error in stage	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"test-cluster","namespace":"grafana"}, "namespace": "grafana", "name": "test-cluster", "reconcileID": "5aa39e3e-d5d3-47fc-848d-c3d15dfbcc3d", "stage": "deployment", "error": "Deployment.apps \"test-cluster-deployment\" is invalid: [spec.template.spec.containers[0].image: Required value, spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey: Required value: can not be empty, spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey: Invalid value: \"\": name part must be non-empty, spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey: Invalid value: \"\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]"}
+grafana_logr_regex  = r'^\s*'
+grafana_logr_regex += r'(\d{4}\-\d{2}\-\d{2}T\d{2}:\d{2}:\d{2}Z)' # Group 1: timestamp
+grafana_logr_regex += r'\s+([A-Z]+)'                              # Group 2: level
+grafana_logr_regex += r'\s+(\S+)'                                 # Group 3: Source
+grafana_logr_regex += r'\s+(.*?)'                                 # Group 4: Message
+grafana_logr_regex += r'\s*$'                                     # Take up any remaining whitespace
 
 def parse_log(line: str) -> dict:
     '''Try to parse the log line with some predefined format
@@ -89,6 +97,10 @@ def parse_log(line: str) -> dict:
         match = re.search(logrus_regex, line)
         log_line['level'] = match.group(2)
         log_line['msg'] = match.group(3)
+    elif re.search(grafana_logr_regex, line) != None:
+        match = re.search(grafana_logr_regex, line)
+        log_line['level'] = match.group(2).lower()
+        log_line['msg'] = match.group(4)
     else:
         try:
             log_line = json.loads(line)