Skip to content

Commit

Permalink
Merge branch 'test' into release
Browse files Browse the repository at this point in the history
  • Loading branch information
René Reitmann committed Nov 16, 2021
2 parents 0d5c943 + 343a241 commit 98a5cc6
Show file tree
Hide file tree
Showing 46 changed files with 463 additions and 176 deletions.
43 changes: 21 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,25 @@ On Windows, `patch.exe` has to exist in the PATH. It is distributed as part of g
## Running on your local machine

Before starting the app on your local machine you need to start the following Document Stores:
1. Mongodb: Mongodb must be running on the default port, on ubuntu you should install it from [here](https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/)
2. Elasticsearch (7.12.1): Elasticsearch must be running on its default port. You can download it from [here](https://www.elastic.co/downloads/elasticsearch)

Make sure that you have read-write-access on the ***data*** directory (in your project directory) for Elasticsearch.
1. Mongodb: Mongodb must be running on the default port, on ubuntu you should install it from [here](https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/). ***However, running it with docker is preferred.***
2. Elasticsearch (7.12.1): Elasticsearch must be running on its default port. You can download it from [here](https://www.elastic.co/downloads/elasticsearch). ***However running it with docker is preferred.***

***Alternatively*** you can run

docker-compose up
# for later use once the containers are created
docker-compose start

to start all services the metadatamanagement depends on. Mongodb and Elasticsearch will be listening on its default ports.
to start all services the metadatamanagement depends on. MongoDB and Elasticsearch will be listening on its default ports. MailDev will show all locally sent email on 8081 and the identity-provider can be setup on port 8082.

Make sure that you have read-write-access on the ***data*** directory (in your project directory) for Elasticsearch.

You can get a MongoDB dump and restore it locally:
```
wget https://metadatamanagement-public.s3.eu-central-1.amazonaws.com/20211019_metadatamanagement.zip
unzip 20211019_metadatamanagement.zip
mongorestore ./metadatamanagement --db=metadatamanagement
```

You will need to setup your `~/.m2/settings.xml` so that maven can download a dependency from Github:

Expand Down Expand Up @@ -71,6 +78,15 @@ auto-refreshes when files change on your hard drive.
mvn
grunt

If you run the backend on your machine for the first time or you have restored a mongodb dump then you need to setup the elasticsearch indices. Therefore go to http://localhost:8080/de/health and sign in with `localuser` and password `herlichwillkommen`. Then click the red button 'reindex'. Reindexing can take up to 1 hour.

In order for all external services to work on your local machine, you need to set the following environment variables:
```
DARA_ENDPOINT=http://labs.da-ra.de/dara/
DARA_USERNAME={see s3://metadatamanagement-private/sensitive_variables.tf}
DARA_PASSWORD={see s3://metadatamanagement-private/sensitive_variables.tf}
```

If you want to build a docker image for the metadatamanagement server app you can run

mvn deploy
Expand All @@ -90,23 +106,6 @@ these new files.

We test our project continuously with the Robot Framework. Test Developers can get further info [here](https://github.com/dzhw/metadatamanagement/wiki/Robot-Framework).

## Technical Documentation

### Domain Model
The following picture models the relationships and attributes of the domain objects which are managed by our system.
![Domain Model](https://github.com/dzhw/metadatamanagement/wiki/images/domain-model.png)

Javadoc for our domain model can be found [here](https://dzhw.github.io/metadatamanagement/).

### Architecture

A (german) overview of the Systemarchitecture can be found [here](https://github.com/dzhw/metadatamanagement/wiki/Architektur).

The following picture gives a rough overview:
![Architecture](https://github.com/dzhw/metadatamanagement/wiki/images/architecture/aws_components_overview.png)

This project is currently built and deployed to AWS Fargate by [Github Actions][GithubActions] (not TravisCI anymore as shown in the picture above). You can test the latest version on [our dev stage.](https://dev.metadata.fdz.dzhw.eu/)

# Big Thanks

Cross-browser Testing Platform and Open Source :heart: Provided by [Sauce Labs][saucelabs]
Expand Down
3 changes: 1 addition & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@
'sphinx.ext.coverage',
'sphinx.ext.mathjax',
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode',
'javasphinx',
'sphinx.ext.viewcode'
]

# Add any paths that contain templates here, relative to this directory.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
javasphinx==0.9.15
docutils==0.17.1
6 changes: 3 additions & 3 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "Metadatamanagement",
"version": "1.0.115",
"version": "1.0.116-SNAPSHOT",
"description": "Data Search for Higher Education Research and Science Studies",
"private": true,
"dependencies": {
Expand Down Expand Up @@ -75,7 +75,7 @@
"grunt-contrib-cssmin": "4.0.0",
"grunt-contrib-htmlmin": "3.1.0",
"grunt-contrib-jshint": "3.1.1",
"grunt-contrib-uglify-es": "git://github.com/gruntjs/grunt-contrib-uglify.git#harmony",
"grunt-contrib-uglify-es": "git+https://github.com/gruntjs/grunt-contrib-uglify.git#harmony",
"grunt-contrib-watch": "1.1.0",
"grunt-html-angular-validate": "0.6.1",
"grunt-jscs": "3.0.1",
Expand Down
12 changes: 6 additions & 6 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.5</version>
<version>2.5.6</version>
<relativePath />
</parent>

<groupId>eu.dzhw.fdz</groupId>
<artifactId>metadatamanagement</artifactId>
<version>1.0.115</version>
<version>1.0.116-SNAPSHOT</version>
<packaging>war</packaging>
<name>metadatamanagement</name>

Expand Down Expand Up @@ -71,7 +71,7 @@
<dependency>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs</artifactId>
<version>4.4.1</version>
<version>4.4.2</version>
<scope>provided</scope>
<optional>true</optional>
</dependency>
Expand Down Expand Up @@ -175,7 +175,7 @@
<dependency>
<groupId>org.javers</groupId>
<artifactId>javers-spring-boot-starter-mongo</artifactId>
<version>6.2.5</version>
<version>6.5.0</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
Expand All @@ -190,7 +190,7 @@
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<version>1.5.11</version>
<version>1.5.12</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
Expand Down Expand Up @@ -302,7 +302,7 @@
<dependency>
<groupId>org.zalando</groupId>
<artifactId>logbook-spring-boot-starter</artifactId>
<version>2.13.0</version>
<version>2.14.0</version>
<exclusions>
<!-- Exclude http client cause it introduces vulnerability issues (snyk errors)-->
<exclusion>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.projection.AnalysisPackageSubDocumentProjection;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation.ValidAnalysisPackageId;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation.UniqueScriptUuids;
import eu.dzhw.fdz.metadatamanagement.common.domain.AbstractShadowableRdcDomainObject;
import eu.dzhw.fdz.metadatamanagement.common.domain.I18nLink;
import eu.dzhw.fdz.metadatamanagement.common.domain.I18nString;
Expand Down Expand Up @@ -54,6 +55,8 @@
+ "eu/dzhw/fdz/metadatamanagement/analysispackagemanagement/domain/"
+ "AnalysisPackage.html'>here</a> for further details.")
@ValidShadowId(message = "analysis-package-management.error.analysis-package.id.pattern")
@UniqueScriptUuids(
message = "analysis-package-management.error.analysis-package.scripts.unique-uuids")
public class AnalysisPackage extends AbstractShadowableRdcDomainObject
implements AnalysisPackageSubDocumentProjection {
private static final long serialVersionUID = 3006197991079331471L;
Expand Down Expand Up @@ -152,9 +155,9 @@ public class AnalysisPackage extends AbstractShadowableRdcDomainObject
@Valid
private List<@I18nStringSize(max = StringLengths.MEDIUM,
message = "analysis-package-management.error.analysis-package.institution.i18n-string-size")
@I18nStringEntireNotEmpty(
@I18nStringEntireNotEmpty(
message = "analysis-package-management.error.analysis-package.institution"
+ ".i18n-string-entire-not-empty") I18nString> institutions;
+ ".i18n-string-entire-not-empty") I18nString> institutions;

/**
* The names of the sponsors which have sponsored the study or project from which this analysis
Expand All @@ -166,9 +169,9 @@ public class AnalysisPackage extends AbstractShadowableRdcDomainObject
@Valid
private List<@I18nStringSize(max = StringLengths.MEDIUM,
message = "analysis-package-management.error.analysis-package.sponsor.i18n-string-size")
@I18nStringEntireNotEmpty(
@I18nStringEntireNotEmpty(
message = "analysis-package-management.error.sponsor.institution"
+ ".i18n-string-entire-not-empty") I18nString> sponsors;
+ ".i18n-string-entire-not-empty") I18nString> sponsors;

/**
* The license of this analysis package. Markdown is supported.
Expand Down Expand Up @@ -201,8 +204,8 @@ public class AnalysisPackage extends AbstractShadowableRdcDomainObject
* At least one {@link Script} must be specified.
*/
@Valid
@Size(min = 1, message = "analysis-package-management.error.analysis-package."
+ "scripts.at-leat-one")
@Size(min = 1,
message = "analysis-package-management.error.analysis-package." + "scripts.at-leat-one")
private List<Script> scripts;

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ public class Script implements Serializable {
* Client side generated id of the script. Used to reference script attachments. Not unique in the
* DB cause shadow copies of {@link AnalysisPackage}s do not changes this id.
*
* Must not be empty.
* Must not be empty and must be unique within the {@link AnalysisPackage}.
*/
@NotEmpty(message = "analysis-package-management.error.script.uuid.not-empty")
private String uuid;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import org.javers.core.metamodel.annotation.Entity;
import org.springframework.data.annotation.Id;

import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation.UniqueScriptId;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation.AtMostOneAttachmentPerScript;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation.ValidScriptUuid;
import eu.dzhw.fdz.metadatamanagement.common.domain.AbstractShadowableRdcDomainObject;
import eu.dzhw.fdz.metadatamanagement.common.domain.util.Patterns;
Expand All @@ -31,8 +31,9 @@
@Data
@AllArgsConstructor
@Builder
@UniqueScriptId(message = "analysis-package-management.error.script-attachment-metadata"
+ ".script-uuid.not-unique")
@AtMostOneAttachmentPerScript(
message = "analysis-package-management.error.script-attachment-metadata"
+ ".script-uuid.not-unique")
@ValidScriptUuid(message = "analysis-package-management.error.script-attachment-metadata"
+ ".script-uuid.not-exists")
public class ScriptAttachmentMetadata extends AbstractShadowableRdcDomainObject {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@
* Ensure that there is at most one attachment per script.
*/
@Documented
@Constraint(validatedBy = {UniqueScriptIdValidator.class})
@Constraint(validatedBy = {AtMostOneAttachmentPerScriptValidator.class})
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface UniqueScriptId {
public @interface AtMostOneAttachmentPerScript {

/**
* Defines the default error message.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
/**
* Validate that there is at most one {@link ScriptAttachmentMetadata} per {@link Script}.
*/
public class UniqueScriptIdValidator
implements ConstraintValidator<UniqueScriptId, ScriptAttachmentMetadata> {
public class AtMostOneAttachmentPerScriptValidator
implements ConstraintValidator<AtMostOneAttachmentPerScript, ScriptAttachmentMetadata> {
@Autowired
private GridFsOperations operations;

Expand All @@ -25,7 +25,7 @@ public class UniqueScriptIdValidator
* @see javax.validation.ConstraintValidator#initialize(java.lang.annotation.Annotation)
*/
@Override
public void initialize(UniqueScriptId constraintAnnotation) {}
public void initialize(AtMostOneAttachmentPerScript constraintAnnotation) {}

/*
* (non-Javadoc)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.validation.Constraint;
import javax.validation.Payload;

import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.AnalysisPackage;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.Script;

/**
* Ensure that all {@link Script} uuids are unique within the {@link AnalysisPackage}.
*
* @author René Reitmann
*/
@Documented
@Constraint(validatedBy = {UniqueScriptUuidsValidator.class})
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface UniqueScriptUuids {

/**
* Defines the default error message.
*/
String message() default "{eu.dzhw.fdz.metadatamanagement.analyispackagemanagement"
+ ".domain.validation.unqiueScriptUuid.message}";

/**
* This contains groups.
*/
Class<?>[] groups() default {};

/**
* This method contains the payload.
*/
Class<? extends Payload>[] payload() default {};

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.validation;

import java.util.List;
import java.util.stream.Collectors;

import javax.validation.ConstraintValidator;
import javax.validation.ConstraintValidatorContext;

import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.AnalysisPackage;
import eu.dzhw.fdz.metadatamanagement.analysispackagemanagement.domain.Script;

/**
* Ensure that all {@link Script} uuids are unique within the {@link AnalysisPackage}.
*
* @author René Reitmann
*/
public class UniqueScriptUuidsValidator
implements ConstraintValidator<UniqueScriptUuids, AnalysisPackage> {

/*
* (non-Javadoc)
*
* @see javax.validation.ConstraintValidator#initialize(java.lang.annotation.Annotation)
*/
@Override
public void initialize(UniqueScriptUuids constraintAnnotation) {}

/*
* (non-Javadoc)
*
* @see javax.validation.ConstraintValidator#isValid(java.lang.Object,
* javax.validation.ConstraintValidatorContext)
*/
@Override
public boolean isValid(AnalysisPackage analysisPackage, ConstraintValidatorContext context) {
if (analysisPackage.getScripts() == null || analysisPackage.getScripts().isEmpty()) {
return true;
}
List<String> uniqueUuids = analysisPackage.getScripts().stream().map(Script::getUuid).distinct()
.collect(Collectors.toList());
return uniqueUuids.size() == analysisPackage.getScripts().size();
}
}
Loading

0 comments on commit 98a5cc6

Please sign in to comment.